Print card with linked object

ABSTRACT

A method of printing content representative of an object on a print medium using a mobile telecommunications device, comprising the steps of: identifying an object using the mobile telecommunications device; receiving the print medium in a media feed path of the mobile telecommunications device; determining a print media identifier from the print medium using a sensor module of the mobile telecommunications device; and, printing content representative of the object on the print medium using a printer module of the mobile telecommunications device.

FIELD OF THE INVENTION

The present invention generally relates to a mobile device incorporatinga printer. The invention more specifically relates to a mobile devicesuch as a mobile telecommunications device, for example a mobile orcellular telephone that incorporates a printer which is able to print awide variety of content on a print medium. However, it will beappreciated by those skilled in the art that the present invention canbe used by other types of portable or mobile devices, or evennon-portable devices.

COPENDING APPLICATIONS

The following applications have been filed by the applicantsimultaneously with the present application:

11/228,540 11/228,500 11/228,501 11/228,530 11/228,531 11/228,50411/228,533 11/228,502 11/228,507 11/228,482 11/228,505 11/228,49711/228,487 11/228,529 11/228,484 11/228,489 11/228,518 11/228,53611/228,496 11/228,488 11/228,506 11/228,516 11/228,526 11/228,53911/228,538 11/228,524 11/228,523 11/228,519 11/228,528 11/228,5277,403,797 11/228,520 11/228,498 11/228,511 11/228,522 11/228,51511/228,537 11/228,534 11/228,491 11/228,499 11/228,509 11/228,49211/228,493 11/228,510 11/228,508 11/228,512 11/228,514 11/228,49411/228,495 11/228,486 11/228,481 11/228,477 7,357,311 7,380,7097,428,986 7,403,796 7,407,092 11/228,513 11/228,503 11/228,48011/228,535 11/228,478

The disclosures of these copending applications are incorporated hereinby reference.

CROSS REFERENCES

The following patents or patent applications filed by the applicant orassignee of the present invention are hereby incorporated bycross-reference:

6,405,055 6,628,430 7,136,186 7,286,260 7,145,689 7,130,075 7,081,9747,177,055 7,209,257 7,161,715 7,154,632 7,158,258 7,148,993 7,075,6847,241,005 7,108,437 6,915,140 6,999,206 7,136,198 7,092,130 6,750,9016,476,863 6,788,336 7,249,108 6,566,858 6,331,946 6,246,970 6,442,5257,346,586 09/505,951 6,374,354 7,246,098 6,816,968 6,757,832 6,334,1906,745,331 7,249,109 7,197,642 7,093,139 10/636,263 10/636,283 10/866,6087,210,038 7,401,223 10/940,653 10/942,858 7,170,652 6,967,750 6,995,8767,099,051 11/107,942 7,193,734 11/209,711 7,095,533 6,914,686 7,161,7097,099,033 7,364,256 7,258,417 7,293,853 7,328,968 7,270,395 11/003,40411/003,419 7,334,864 7,255,419 7,284,819 7,229,148 7,258,416 7,273,2637,270,393 6,984,017 7,347,526 7,357,477 11/003,463 7,364,255 7,357,47611/003,614 7,284,820 7,341,328 7,246,875 7,322,669 10/815,621 7,243,83510/815,630 10/815,637 10/815,638 7,251,050 10/815,642 7,097,0947,137,549 10/815,618 7,156,292 10/815,635 7,357,323 10/815,634 7,137,5667,131,596 7,128,265 7,207,485 7,197,374 7,175,089 10/815,617 10/815,6207,178,719 10/815,613 7,207,483 7,296,737 7,270,266 10/815,614 10/815,6367,128,270 11/041,650 11/041,651 11/041,652 11/041,649 11/041,61011/041,609 11/041,626 11/041,627 11/041,624 7,395,963 11/041,55611/041,580 11/041,723 11/041,698 11/041,648 10/815,609 7,150,3987,159,777 10/815,610 7,188,769 7,097,106 7,070,110 7,243,849 6,623,1016,406,129 6,505,916 6,457,809 6,550,895 6,457,812 7,152,962 6,428,1337,204,941 7,282,164 10/815,628 7,278,727 7,417,141 10/913,374 7,367,6657,138,391 7,153,956 7,423,145 10/913,379 10/913,376 7,122,076 7,148,34511/172,816 11/172,815 11/172,814 7,416,280 7,252,366 10/683,0647,360,865 11/124,158 11/124,196 11/124,199 11/124,162 11/124,20211/124,197 11/124,154 11/124,198 7,284,921 11/124,151 7,407,25711/124,192 11/124,175 7,392,950 11/124,149 7,360,880 11/124,17311/124,155 7,236,271 11/124,174 11/124,194 11/124,164 11/124,20011/124,195 11/124,166 11/124,150 11/124,172 11/124,165 11/124,18611/124,185 11/124,184 11/124,182 11/124,201 11/124,171 11/124,18111/124,161 11/124,156 11/124,191 11/124,159 7,370,932 7,404,61611/124,187 11/124,189 11/124,190 11/124,180 11/124,193 11/124,18311/124,178 11/124,177 11/124,148 7,431,449 11/124,167 11/124,17911/124,169 11/187,976 11/188,011 11/188,014 6,746,105 7,156,5087,159,972 7,083,271 7,165,834 7,080,894 7,201,469 7,090,336 7,156,4897,413,283 10/760,246 7,083,257 7,258,422 7,255,423 7,219,980 10/760,2537,416,274 7,367,649 7,118,192 10/760,194 7,322,672 7,077,505 7,198,3547,077,504 10/760,189 7,198,355 7,401,894 7,322,676 7,152,959 7,213,9067,178,901 7,222,938 7,108,353 7,104,629 7,246,886 7,128,400 7,108,3556,991,322 7,287,836 7,118,197 10/728,784 7,364,269 7,077,493 6,962,40210/728,803 7,147,308 10/728,779 7,118,198 7,168,790 7,172,270 7,229,1556,830,318 7,195,342 7,175,261 10/773,183 7,108,356 7,118,202 10/773,1867,134,744 10/773,185 7,134,743 7,182,439 7,210,768 10/773,187 7,134,7457,156,484 7,118,201 7,111,926 7,431,433 7,018,021 7,401,901 11/060,80511/188,017 11/097,308 11/097,309 7,246,876 7,431,431 7,419,249 7,377,6237,328,978 7,334,876 7,147,306 7,156,289 7,178,718 7,225,979 11/084,79611/084,742 11/084,806 09/575,197 7,079,712 6,825,945 7,330,974 6,813,0397,190,474 6,987,506 6,824,044 7,038,797 6,980,318 6,816,274 7,102,7727,350,236 6,681,045 6,678,499 6,679,420 6,963,845 6,976,220 6,728,0007,110,126 7,173,722 6,976,035 6,813,558 6,766,942 6,965,454 6,995,8597,088,459 6,720,985 7,286,113 6,922,779 6,978,019 6,847,883 7,131,0587,295,839 7,406,445 09/693,690 6,959,298 6,973,450 7,150,404 6,965,8827,233,924 09/575,181 09/722,174 7,175,079 7,162,259 6,718,061 10/291,52310/291,471 7,012,710 6,825,956 10/291,481 7,222,098 10/291,825 7,263,5087,031,010 6,972,864 6,862,105 7,009,738 6,989,911 6,982,807 10/291,5766,829,387 6,714,678 6,644,545 6,609,653 6,651,879 10/291,555 7,293,24010/291,592 7,415,668 7,044,363 7,004,390 6,867,880 7,034,953 6,987,5817,216,224 10/291,821 7,162,269 7,162,222 7,290,210 7,293,233 7,293,2346,850,931 6,865,570 6,847,961 10/685,523 10/685,583 7,162,442 10/685,5847,159,784 10/804,034 7,404,144 6,889,896 10/831,232 7,174,056 6,996,2747,162,088 7,388,985 7,417,759 7,362,463 7,259,884 10/944,043 7,167,2707,388,685 6,986,459 10/954,170 7,181,448 10/981,626 10/981,616 7,324,9897,231,293 7,174,329 7,369,261 7,295,922 7,200,591 11/020,106 11/020,26011/020,321 11/020,319 11/026,045 7,347,357 11/051,032 7,382,48211/107,944 11/107,941 11/082,940 11/082,815 7,389,423 7,401,2276,991,153 6,991,154 11/124,256 11/123,136 11/154,676 7,322,52411/182,002 11/202,251 11/202,252 11/202,253 7,408,670 11/202,21811/206,778 11/203,424 11/222,977 7,068,382 7,007,851 6,957,921 6,457,8837,044,381 7,094,910 7,091,344 7,122,685 7,038,066 7,099,019 7,062,6516,789,194 6,789,191 10/900,129 7,278,018 7,360,089 10/982,975 10/983,0296,644,642 6,502,614 6,622,999 6,669,385 6,827,116 7,011,128 7,416,0096,549,935 6,987,573 6,727,996 6,591,884 6,439,706 6,760,119 7,295,3327,064,851 6,826,547 6,290,349 6,428,155 6,785,016 6,831,682 6,741,8716,927,871 6,980,306 6,965,439 6,840,606 7,036,918 6,977,746 6,970,2647,068,389 7,093,991 7,190,491 10/901,154 10/932,044 10/962,412 7,177,0547,364,282 10/965,733 10/965,933 10/974,742 10/982,974 7,180,60910/986,375 11/107,817 7,292,363 11/149,160 6,982,798 6,870,966 6,822,6396,474,888 6,627,870 6,724,374 6,788,982 7,263,270 6,788,293 6,946,6726,737,591 7,091,960 7,369,265 6,792,165 7,105,753 6,795,593 6,980,7046,768,821 7,132,612 7,041,916 6,797,895 7,015,901 7,289,882 7,148,64410/778,056 10/778,058 10/778,060 10/778,059 10/778,063 10/778,06210/778,061 10/778,057 7,096,199 7,286,887 7,400,937 10/917,466 7,324,8597,218,978 7,245,294 7,277,085 7,187,370 10/917,436 10/943,856 10/919,3797,019,319 10/943,878 10/943,849 7,043,096 7,148,499 11/144,84011/155,556 11/155,557 11/193,481 11/193,435 11/193,482 11/193,4797,055,739 7,233,320 6,830,196 6,832,717 7,182,247 7,120,853 7,082,5626,843,420 10/291,718 6,789,731 7,057,608 6,766,944 6,766,945 7,289,1037,412,651 7,299,969 7,264,173 10/409,864 7,108,192 10/537,159 7,111,7917,077,333 6,983,878 10/786,631 7,134,598 7,431,219 6,929,186 6,994,2647,017,826 7,014,123 7,134,601 7,150,396 10/971,146 7,017,823 7,025,2767,284,701 7,080,780 7,376,884 10/492,169 10/492,152 7,359,551 10/492,1617,308,148 10/502,575 10/531,229 10/531,733 10/683,040 10/510,39110/510,392 10/778,090 6,957,768 09/575,172 7,170,499 7,106,888 7,123,2396,982,701 6,982,703 7,227,527 6,786,397 6,947,027 6,975,299 7,139,4317,048,178 7,118,025 6,839,053 7,015,900 7,010,147 7,133,557 6,914,59310/291,546 6,938,826 7,278,566 7,123,245 6,992,662 7,190,346 7,417,62911/074,782 7,382,354 11/075,917 7,221,781 11/102,843 7,263,225 10/72718110/727,162 7,377,608 7,399,043 7,121,639 7,165,824 7,152,942 10/727,1577,181,572 7,096,137 7,302,592 7,278,034 7,188,282 10/727,159 10/727,18010/727,179 10/727,192 10/727,274 10/727,164 10/727,161 10/727,19810/727,158 10/754,536 10/754,938 10/727,160 10/934,720 7,171,3237,369,270 6,795,215 7,070,098 7,154,638 6,805,419 6,859,289 6,977,7516,398,332 6,394,573 6,622,923 6,747,760 6,921,144 10/884,881 7,092,1127,192,106 11/039,866 7,173,739 6,986,560 7,008,033 11/148,237 7,195,3287,182,422 7,374,266 7,427,117 10/854,488 7,281,330 10/854,503 7,328,95610/854,509 7,188,928 7,093,989 7,377,609 10/854,495 10/854,49810/854,511 7,390,071 10/854,525 10/854,526 10/854,516 7,252,35310/854,515 7,267,417 10/854,505 10/854,493 7,275,805 7,314,26110/854,490 7,281,777 7,290,852 10/854,528 10/854,523 10/854,52710/854,524 10/854,520 10/854,514 10/854,519 10/854,513 10/854,49910/854,501 7,266,661 7,243,193 10/854,518 10/854,517 10/934,6287,163,345 10/760,254 7,425,050 7,364,263 7,201,468 7,360,868 10/760,2497,234,802 7,303,255 7,287,846 7,156,511 10/760,264 7,258,432 7,097,29110/760,222 10/760,248 7,083,273 7,367,647 7,374,355 10/760,20410/760,205 10/760,206 10/760,267 10/760,270 7,198,352 7,364,2647,303,251 7,201,470 7,121,655 7,293,861 7,232,208 7,328,985 7,344,2327,083,272 11/014,764 11/014,763 7,331,663 7,360,861 7,328,973 7,427,1217,407,262 7,303,252 7,249,822 11/014,762 7,311,382 7,360,860 7,364,2577,390,075 7,350,896 7,429,096 7,384,135 7,331,660 7,416,287 11/014,7377,322,684 7,322,685 7,311,381 7,270,405 7,303,268 11/014,735 7,399,0727,393,076 11/014,750 11/014,749 7,249,833 11/014,769 11/014,7297,331,661 11/014,733 7,300,140 7,357,492 7,357,493 11/014,766 7,380,9027,284,816 7,284,845 7,255,430 7,390,080 7,328,984 7,350,913 7,322,6717,380,910 7,431,424 11/014,716 11/014,732 7,347,534 11/097,26811/097,185 7,367,650 6,454,482 6,808,330 6,527,365 6,474,773 6,550,9977,093,923 6,957,923 7,131,724 7,396,177 7,168,867 7,125,098 7,413,3637,249,901 7,188,930 7,079,292

BACKGROUND OF THE INVENTION

The assignee has developed mobile or cellular telephones, Personal DataAssistants (PDAs) and other mobile telecommunication devices, with theability to print hard copies of content, such as images or informationstored or accessed by the device, (see for example, U.S. Pat. No.6,405,055, filed on Nov. 9, 1999). Likewise, the assignee has alsodesigned digital cameras with the ability to print captured images withan in-built printer (see for example, U.S. Pat. No. 6,750,901, filed onJul. 10, 1998). As the prevalence of mobile telecommunications devicesincreases, the functionality of these devices is further enhanced by theability to print hard copies.

As these devices are portable, they should be compact for userconvenience. Accordingly, any printer incorporated into the device needsto maintain a small form factor. Also, the additional load on thebattery should be relatively small. Furthermore, the consumables (suchas ink, paper, etc.) should be relatively inexpensive and simple toreplenish. It is these factors that strongly influence the commercialsuccess or otherwise of products of this type.

The assignee of the present invention has also developed the Netpagesystem for enabling interaction with computer software using a printedinterface and a proprietary stylus-shaped sensing device. As describedin detail in U.S. Pat. No. 6,792,165, filed on Nov. 25, 2000 and U.S.patent application Ser. No. 10/778,056, filed on Feb. 17, 2004, aNetpage pen captures, identifies and decodes tags of coded data printedonto a surface such as a page. In a preferred Netpage implementation,each tag encodes a position and an identity of the document. By decodingat least one of the tags and transmitting the position (or a refinedversion of the position, representing a higher resolution position ofthe pen) and identity referred to by the decoded tag, a remote computercan determine an action to perform. Such actions can include, forexample, causing information to be saved remotely for subsequentretrieval, downloading of a webpage for printing or display via acomputer, bill payment or even the performance of handwritingrecognition based on a series of locations of the Netpage pen relativeto the surface.

When printing a Netpage, a printer in a mobile telecommunications devicecan print the Netpage tags simultaneously with visible user information.The association between the tags and information can already exist on aremote Netpage server, such as where the printer is printing a fullyrendered page (including tags) provided by the Netpage server or anothercomputer. Alternatively, the mobile telecommunications device cangenerate the tags (or source them remotely) and define an associationbetween the tags and user information. The association is then recordedin the remote Netpage server.

A problem with these options is that they require the mobiletelecommunications device to include Netpage tag printing capabilities.This requires an additional row of print nozzles in the printhead, andreduces the amounts of ink that can be stored for non-tag use. Whilstthis is less of an issue with large, mains-powered printers, it can bean issue in small form-factor articles such as mobile telecommunicationsdevices. Alternatively, the mobile telecommunications device can beconfigured to print on print media that is pre-printed with Netpagetags. That way the printer need only print the user information andrecord an association between the visible information and thepre-printed tags.

It is desirable to provide functional applications making use of themobile telecommunications device. Such applications can include, forexample, mobile printing applications, linking, capturing and/orprinting generic or specific objects to a print medium, and many otherapplications providing functionality to the mobile telecommunicationsdevice and various uses of types of print media.

SUMMARY OF THE INVENTION

In one particular, but non-limiting, aspect, an M-Print device is amobile device such as a telephone or PDA which incorporates a printer.Paper is either manually presented or auto-fed from a cartridge,depending on device form factor. The printer may or may not print tags,for example infrared tags, and the printer or a sensor detects tagsprinted, for example pre-printed, onto blank media. The paper patheither includes a tag reader, or it includes a simpler sensor forreading a linear data track on the card. The data track can encode thesame identifier as the tags. Reading the identifier allows the M-Printdevice to associate the card's graphic and/or interactive content withthe identifier. This allows subsequent interactions with the card to beproperly interpreted. The graphic and/or interactive content is storedon a network-based server, indexed by the identifier.

It should be noted that the media identifier (i.e. print mediaidentifier) may correspond to a range of 2D coordinates without anexplicit single media identifier. Hence, reference to the mediaidentifier is to be read as a reference to an explicit or defined one ormore media identifiers, or, as a reference to a range of 2D coordinates.

The device also optionally incorporates a pointer. The pointer may beused to click on a hyperlink, but generally doesn't operate at asufficiently high rate to capture motion. Alternatively, the telephonemay incorporate a fully-functional Netpage-type pen. Even when theM-Print device doesn't incorporate a pointer, the user can interact withprinted cards by feeding them through the paper path. The data trackreader or tag reader in the paper path extracts the identifier, whichallows the device to identify the graphic and/or interactive content ofthe card, and object(s) linked to the card. Not all M-Print cards haveto be produced by an M-Print device. For example, pre-printed M-Printcards of a collectible or promotional nature may be included in cerealpackets or magazines. And even blank media may bear advertising on thereverse side. Not all M-Print cards have to be interacted with via apointer in an M-Print device. They can be interacted with via anydevice, or another scanning device altogether which can read the datatrack or an application-specific printed barcode.

An M-Print card acts as a token for the graphic and/or interactivecontent of the card, including any objects linked to the card. A usercan easily obtain the original digital content of the card by clickingon the card or ‘virtually scanning’ the card through the paper path. Forexample, a photo acts as a token for the original digital image, and abusiness card acts as a token for the contact details linked to thecard. By acting as a token for its own content, a card allows a user toobtain a perfect re-print. In addition to the identifier, the data trackand the tags encode a digital signature which allows the card to beauthenticated. This has two purposes. Firstly, it allows a blank card tobe authenticated during printing to prevent the use of non-sanctionedblanks. Secondly, it allows a card to be authenticated when used as atoken, to prevent fraudulent access to the content of the card orobjects linked to the card.

Various applications are possible using aspects, components or featuresof the mobile telecommunications device and associated coded printmedium. Such applications can include mobile printing applications,linking, capturing and/or printing generic or specific objects to aprint medium, and many other applications providing practical uses forthe coded print medium and/or the mobile telecommunications device.Various particular applications are herein described.

In a first aspect the present invention provides a method of printingcontent representative of an object on a print medium using a mobiletelecommunications device, comprising the steps of:

-   -   identifying an object using the mobile telecommunications        device;    -   receiving the print medium in a media feed path of the mobile        telecommunications device;    -   determining a print media identifier from the print medium using        a sensor module of the mobile telecommunications device; and,    -   printing content representative of the object on the print        medium using a printer module of the mobile telecommunications        device.

Optionally the content includes one or more interactive elements.

Optionally the object is obtained from a server in communication withthe mobile telecommunications device.

Optionally the object is an image and is captured using a camera of themobile telecommunications device.

Optionally the print medium is provided with first coded data in a firstdata format, the first coded data encoding first information.

Optionally the first information includes the print media identifier.

Optionally the first format is a linear pattern.

Optionally the print medium is provided with second coded data in asecond data format, the second coded data encoding second information.

Optionally the second information is indicative of a two-dimensionalcoordinate grid.

Optionally the second information includes the print media identifier.

Optionally the second information is indicative of regions of the printmedium surface.

Optionally the printer module prints the second coded data on the printmedium.

Optionally the second coded data is pre-printed on the print mediumprior to inserting the print medium.

Optionally the object is a linked object and is associated with a regionof the print medium using the second coded data and the linked object isstored in an object repository.

Optionally the printer module prints the first coded data on the printmedium.

Optionally the object is statically associated with a document.

Optionally the object is dynamically associated with a document.

Optionally the object is associated with a particular location on theprint medium.

Optionally the object is associated with a form, as a particular fieldof a form or a particular instance of a form.

Optionally different access permissions can be associated with theobject when stored.

BRIEF DESCRIPTION OF THE DRAWINGS

An example embodiment of the present invention should become apparentfrom the following description, which is given by way of example only,of a preferred but non-limiting embodiment, described in connection withthe accompanying figures.

FIG. 1 illustrates an example High Level Architecture;

FIG. 2 illustrates example M-Doc Retriever Components;

FIG. 3 illustrates an example Nugget Generation Service;

FIG. 4 illustrates an example Player;

FIG. 5 illustrates an example PlayRequest;

FIG. 6 illustrates example Values, Types and Categories;

FIG. 7 illustrates an example interactive business card;

FIG. 8 illustrates an example Player Sequence;

FIG. 9 illustrates an example RequestRouter and related classes;

FIG. 10 illustrates an example UserRequestRouter;

FIG. 11 illustrates a typical arrangement of routers and player agentsin a Netpage system;

FIG. 12 illustrates example Capability and Request Propagation;

FIG. 13 illustrates an example Capability Aggregation;

FIG. 14 illustrates an example Capability Transformation;

FIG. 15 illustrates example PlayerProfiles;

FIG. 16 illustrates an example Printed Interface for selecting thecurrent player profile;

FIG. 17 illustrates example PlayRequests embedded in an interactivedocument;

FIG. 18 illustrates an example Request Routing;

FIG. 19 illustrates an example Synchronous Messaging Sequence Diagram;

FIG. 20 illustrates an example Asynchronous Messaging CommunicationSequence;

FIG. 21 illustrates an example Streaming Messaging Sequence;

FIG. 22 illustrates an example Interactive Messaging Sequence;

FIG. 23 illustrates an example Hybrid Messaging Sequence;

FIG. 24 illustrates an example Player Session Sequence Diagram;

FIG. 25 illustrates an example Player Session Detailed Sequence Diagram;

FIG. 26 illustrates an example Desktop Player Deployment;

FIG. 27 illustrates an example Short-Range Thin Mobile PlayerDeployment;

FIG. 28 illustrates an example Long-Range Thin Mobile Player Deployment;

FIG. 29 illustrates an example Smart Mobile Player;

FIG. 30 illustrates an Object association being displayed in a physicalPlayer Device;

FIG. 31 illustrates an Object association being displayed in theExplorer application;

FIG. 32 illustrates a Creation of an impression object association;

FIG. 33 illustrates an example Tagged Sticker;

FIG. 34 illustrates an example Reusable Sticker;

FIG. 35 illustrates an example Sticker with “Confirm Action”;

FIG. 36 illustrates an example Sticker with limited interactive areas toprevent accidental invocation of destructive operations;

FIG. 37 illustrates example Category Specific Stickers;

FIG. 38 illustrates an example Swipe based printed toolbar for creatingimpression associations;

FIG. 39 illustrates an example Swipe Based Sticker with TransparentRegion;

FIG. 40 illustrates an example Swipe Based Sticker with Graphics overthe Transparent Region;

FIG. 41 illustrates an example Impression Associations Object Model;

FIG. 42 illustrates example Field Associations in the Netpage Server;

FIG. 43 illustrates an example Object Association Sample Application;

FIG. 44 illustrates an example Underlying Form;

FIG. 45 illustrates an example Overlayed Form;

FIG. 46 illustrates an example Printed Contacts with Phone Numbers;

FIG. 47 illustrates an example State machine for basic clipboardinteraction;

FIG. 48 illustrates an example interactive Netpage card with commonoperations;

FIG. 49 illustrates an example State machine for operation basedclipboard;

FIG. 50 illustrates simultaneously supporting both object first andcommand first models;

FIG. 51 illustrates example Single use clipboard entries;

FIG. 52 illustrates an example Single-use clipboard with timeouts;

FIG. 53 illustrates an example Multi-use clipboard with timeouts;

FIG. 54 illustrates an example Interactive card for selecting a printer;

FIG. 55 illustrates example Field details of printer selection card;

FIG. 56 illustrates an example Netpage form containing various commands;

FIG. 57 illustrates an example Command form showing details of officeprinter field;

FIG. 58 illustrates an example SMS Based Downloadable Content Purchase;

FIG. 59 illustrates an example Netpage Play Sequence for Previewing aRingtone;

FIG. 60 illustrates an example Using play requests to deliver theproduct;

FIG. 61 illustrates an example Using play requests to purchase theproduct, and traditional delivery;

FIG. 62 illustrates an example Hybrid approach using traditionaldelivery on the last hop to the handset;

FIG. 63 illustrates an example Load card;

FIG. 64 illustrates an example Validating an ID;

FIG. 65 illustrates an example High level printing sequence;

FIG. 66 illustrates an example High level sequence diagram for uploadingfrom a mobile device;

FIG. 67 illustrates an example High level sequence diagram fordownloading to a mobile device, using a SMS alert to trigger thedownload;

FIG. 68 illustrates an example Sequence fragment showing the processingof scanned ID;

FIG. 69 illustrates an example Local Photo Printing Sequence;

FIG. 70 illustrates an example Printing Uploads to a Photo Archive;

FIG. 71 illustrates an example Capturing a Netpage document viaprinting;

FIG. 72 illustrates an example Business Card;

FIG. 73 illustrates example Business Card Phone Number Fields;

FIG. 74 illustrates an example Business Card Fax Field;

FIG. 75 illustrates an example Business Card Web URL Field;

FIG. 76 illustrates example Business Card SMS and MMS Fields;

FIG. 77 illustrates an example Business Card Email Field;

FIG. 78 illustrates an example Street Address Field;

FIG. 79 illustrates an example Business Card Photo and Name Field;

FIG. 80 illustrates an example Business Card Identifier Field;

FIG. 81 illustrates an example Printed photo card;

FIG. 82 illustrates example Interactive fields for photo card;

FIG. 83 illustrates an example Scanning of an M-Print printout;

FIG. 84 illustrates an example Sequence diagram for generating Netpageclicks from a mobile device GUI;

FIG. 85 illustrates a schematic representation of the modularinteraction in a printer/mobile phone;

FIG. 86 illustrates a schematic representation of the modularinteraction in a tag sensor/mobile phone;

FIG. 87 illustrates a schematic representation of the modularinteraction in a printer/tag sensor/mobile phone;

FIG. 88 is a more detailed schematic representation of the architecturewithin the mobile phone of FIG. 87;

FIG. 89 is a more detailed schematic representation of the architecturewithin the mobile phone module of FIG. 88;

FIG. 90 is a more detailed schematic representation of the architecturewithin the printer module of FIG. 88;

FIG. 91 is a more detailed schematic representation of the architecturewithin the tag sensor module of FIG. 88;

FIG. 92 is a schematic representation of the architecture within a tagdecoder module for use instead of the tag sensor module of FIG. 88;

FIG. 93 illustrates an exploded perspective view of a “candy bar” typemobile phone embodiment;

FIG. 94 illustrates a partially cut away front and bottom view of theembodiment shown in FIG. 93;

FIG. 95 illustrates a partially cut away rear and bottom view of theembodiment shown in FIG. 93;

FIG. 96 illustrates a front elevation of the embodiment shown in FIG. 93with a card being fed into the entry slot;

FIG. 97 illustrates a cross section view taken along line A-A of FIG.96;

FIG. 98 illustrates a cross section view taken along line A-A of FIG. 96with the card emerging from the media exit slot of the mobile phone;

FIG. 99 illustrates a lateral cross section through a print cartridge;

FIG. 100 illustrates the media coding on the card with separate clockand data tracks;

FIG. 101 illustrates a block diagram of an M-print system that usesmedia with separate clock and data tracks;

FIG. 102 illustrates a simplified circuit diagram for an opticalencoder;

FIG. 103 illustrates a block diagram of the MoPEC with the clock anddata inputs;

FIG. 104 illustrates a block diagram of the optional edge detector andpage sync generator for the M-print system of FIG. 101;

FIG. 105 illustrates a block diagram of a MoPEC that uses media with apilot sequence in the data track to generate a page sync signal;

FIG. 106 illustrates a schematic representation of the position of theencoders along the media feed path.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following modes, given by way of example only, are described inorder to provide a more precise understanding of the subject matter of apreferred embodiment or embodiments. In the figures, incorporated toillustrate features of an example embodiment, like reference numeralsare used to identify like parts throughout the figures.

1.0 Printing Internet Based Content Product Architecture

An example of a M-Print print media is the size of a business card. Ingeneral, documents or web based materials that have been designed fordisplay on a desktop monitor or to be printed on A4 or Letter paper andmay not print well on such sized media. If the content is reduced forthe business card media then the content may be too small for easyreading. If multiple pages of media are used to print a page then theuser is required to assemble the pages in the correct order before theprintout is meaningful. To have presentable, effective content on abusiness card sized media the content should be specifically authoredfor that sized media. Described herein is a general mechanism to allowthe authors and providers of web applications and web sites to makeexplicit use of the new media size, such as a business card.

The term “Mobile Document” or M-Doc is herein used to refer to documentsspecifically authored to be printed via M-Print. The format of a “MobileDocument” (i.e print medium) may vary, it can be pre-rendered and in aformat ready to be sent directly to the printer, or it can be in ahigher level format that requires rendering before printing. On somemobile devices it is not be possible to render the “Mobile Document” onthe device, thus the “Mobile Document” is rendered before being sent tothe mobile device. In regard of other mobile devices, the “MobileDocument” can be sent in the high level format and rendered on themobile device. In general, to be able to render on the device the“Mobile Document” format is provided in an encapsulated format thatcontains the data necessary to render the M-Doc. Thus, by providing theM-Doc in an encapsulated format, the M-Doc does not necessarily have torely on a particular font or bitmap being available on the mobile device(i.e. mobile telecommunications device).

A common usage of a “Mobile Document” is for the author of a web page tosummarise contents of the web-page in a “Mobile Document”, which willherein be referred to as a “Nugget”, and to provide a link on the pagefor users to print the nugget. When the web page is static HTML thecontent of the nugget can also be static. If the web page is dynamicHTML then it is likely the content of the nugget may also have to bedynamically created.

Current 2.5G mobile data networks have low bandwidth, high latency, andare expensive to transfer data over. The emerging 3G networks improvethe bandwidth and latency but are still expensive to transfer data over.Mobile carriers tend to subsidise some of the data transfer mechanismsto encourage use, so it is possible to have a situation where it issignificantly cheaper to send data via an MMS than it is to transfer itvia a HTTP request over the same network. For this reason the proposedarchitecture supports multiple ways of delivering an M-Doc to a mobiledevice. The architecture is also designed to minimise the number ofrequests that need to be made from the device to retrieve a M-Doc and toalso to minimise the amount of data that needs to be transferred to thedevice to transmit an M-Doc.

There are three messaging services in common use in the mobile networksat the moment: SMS (Short Message Service), EMS (Enhanced MessageService and MMS) (Multimedia Message Service). SMS is generally designedfor sending text only messages up to 160 characters long. EMS is anenhanced version of SMS consisting of several SMS messages clusteredtogether. This mechanism is used to deliver ring tones, etc to handsets.Both SMS and EMS are implemented using existing mechanisms in the GSM orCDMA networks and do not require IP based bearers such as GPRS. MMSprovides the ability to send a mixture of multimedia formats such asimages, sound and movies along with a definition of how to use thesemultimedia formats using the Synchronized Multimedia IntegrationLanguage, SMIL. MMS does not have any theoretical size limits. MMS isimplemented on top of IP bearers and requires GPRS or one the 3Gequivalents to be deployed. Since MMS uses IP bearers it is not able toself-transfer or “push” itself out to a handset, instead it uses SMS toalert the user to the fact that an MMS message is waiting in the networkfor the user to retrieve, so it requires a “push” and “pull” to retrievethe message, whereas SMS requires a single “push” and EMS is deliveredby multiple “pushes”.

1.1 High Level Architecture

Referring to FIG. 1, a M-Doc 500 residing in the network 501 generallyrequires delivery to the mobile device 100 before it can be printed.There are two ways in which a request to retrieve and print an M-Doc 500can originate:

-   -   1. direct user action, typically clicking on a link on a web        page; or    -   2. the arrival of an SMS or MMS containing a M-Doc reference 507        or an M-Doc 500 itself.

A M-Doc Retriever 502 is a component responsible for fetching an M-Doc500. The M-Doc 500 is then passed to the M-Doc Printing Service 503 andprinted. The following sections explain each of these major componentsand their inputs and outputs in more detail.

1.1.1 Web/WAP Browser

The Web Browser 504 is a third party application available on the mobiledevice 100. It is used by the user to browse web pages. A web site thatsupports printing M-Docs 500 includes web pages that contain M-DocReference links. When the user clicks on a M-Doc Reference link, a M-Doc500 reference is returned to the browser 504. The M-Doc reference 507can be handled by the browser 504 in a number of ways dependent upon theoperating system running on the mobile device 100.

It is also possible for the M-Doc Retriever 502 to be activated directlyby passing the M-Doc Retriever 502 a web page reference. This triggers aNugget Creation service 506 to generate a Nugget for the website whichthe web page reference is associated with. The ability to generate ameaningful Nugget for a website depends on the content of the website.

1.1.2 M-Doc Retriever

The M-Doc retriever 502 is activated by the arrival of M-Doc reference507 to the device.

Referring to FIG. 2, the M-Doc Retriever 502 is responsible for taking aM-Doc Reference 507 and resolving it to an M-Doc 500 to be passed ontothe M-Doc Printing Service 503. There are a number of ways that a M-DocReference 507 can be supplied to the M-Doc Retriever 502:

-   -   1. MIME type recogniser 508—This can be activated either by        clicking on a link in a web page that causes an “HTTP Get” of an        object whose MIME type is an M-Doc reference 507. Or it can be        activated by the arrival on the mobile device of an Obex        transfer 511.    -   2. Message Monitor 509—This is a component that monitors the        Inbox of the Messaging Service on the mobile device. When it        receives a message from any source that contains an M-Doc        reference 507, the Message Monitor 504 passes the M-Doc        reference 507 onto the M-Doc Retriever 502.    -   3. Web browser plugin 510—This is triggered by explicit        scripting code in a web page being browsed on a web browser 504.        When the Web browser plugin is activated, it is passed a M-Doc        reference 507 which is passed onto the M-Doc Retriever 502.

The M-Doc Retriever 502 receives the M-Doc Reference 507 which is in theformat of a URI (Uniform Resource Identifier). The M-Doc Retriever 502appends device specific information to URI and is dispatched via an HTTPrequest to the M-Doc Retrieval Service 512 running on the device. Devicespecific information that is appended to the URI is dependent upon howthe system has been deployed and the capabilities of the device.

If a Mobile Device Capability Service 513 is deployed in the network 501then the device specific information appended only requiresidentification the handset, via IMEI. If the Mobile Device CapabilityService 513 is not deployed in the network 501 then the device 100 isrequired to append information relating to the printer 4, the formats ofM-Doc's 500 the device 100 is capable of printing and the preferreddelivery method. The printer information required is deploymentspecific. If the renderer is able to look up the printer characteristicsbased on the handset or a printer version number then only those need tobe provided, but if not, then information relating the printersresolution and colour space parameters are required.

The M-Doc URI that is supplied by the application is used to retrievethe M-Doc 500. This URI contains information needed to retrieve orgenerate the M-Doc 500 from the application 514. If the format of thereturned M-Doc 500 matches the format(s) the device 100 is capable ofprinting, then the document 500 is delivered to the device 100. If thereis an unsuccessful match, the M-Doc 500 is rendered. This is performedby passing the document 500 to the rendering service 515 along with theprinter information. The rendering service returns the document in apre-rendered format which can be printed by a mobile device 100containing a printer 4.

The way in which the M-Doc 500 is delivered back to the M-Doc retriever502 can also vary. It may be returned in the reply to the original HTTPrequest or it may be sent via an MMS or e-mail. These later two casesare most likely to be used in an environment where the pricing policy ofa carrier encourages MMS or email use over general web browsing. Thepreferred delivery method may be included in the retrieval request or itmay be looked up via the Mobile Device Capability Service 513.

The Rendering Service 515 may also be used directly by applicationwriters who want to provide pre-rendered M-Doc's within theirapplication. In this case the Rendering Service 515 is accessed via SOAP(Simple Object Access Protocol) as a web service, providing the M-Doc500 in its authored format and obtaining the pre-rendered print formatdocument and a thumbnail image for use in a GUI 516 of the application514 running on the device 100.

When an MMS 517 is sent to the mobile device 100, the MMS 517 is storedin the mobile network at a MMS Message Centre 518 and an SMS is sent tothe device to alert the user an MMS 517 is waiting to be fetched. With amodification this notification mechanism can be used to deliver an M-Doc500 to a phone 100. The SMS notification can contain both an M-Docreference 507 and an MMS notification. The request to fetch the MMS fromthe Message Centre 518 can be enhanced with the M-Doc reference 509,allowing the Message Centre 518 to contact the M-Doc Retrieval Service512 to retrieve the M-Doc 500 in the body of the MMS 517. This serviceis called an M-Doc MMS service.

Any email, MMS, Obex or web page may contain a direct M-Doc 500 ratherthan an M-Doc Reference 507. In this case the M-Doc 500 is passeddirectly onto the M-Doc Printing Service 503. Unless the sender of themessage knows the capability of the handset receiving the M-Doc 500, theM-Doc 500 may not be able to be rendered appropriately, hence the M-DocReference 507 approach is preferred, but in some cases, such as asubscription, the sender may know the capabilities of the handset andhence be able to by-pass the M-Doc Retrieval process and deliver theM-Doc 500 directly.

1.1.3 M-Doc Printing Service

The M-Doc Printing Service 503 prints the document 500 to the printer 4in the mobile device 100. M-Doc's 500 may have different documentformats, but the M-Doc Retriever stage ensures that it retrieves anM-Doc 500 in a format that can be printed by the mobile device 100without any further network interactions.

1.1.4 Nugget Production

A Nugget is the distillation of the content of a web page onto anM-Print sized printout. In general this not only involves reducing theweb layout so that the content fits onto the M-Print printout. Theprocess also involves selecting the key pieces of information on the webpage and explicitly composing an M-Doc 500 that presents the informationappropriately. Nugget support can be provided in two ways:

-   -   1. As part of the design of the application 514    -   2. By a nugget generating service 519.

Providing nugget support as part of the design of web application 514requires the author of the web interface to provide a link to a nuggeton the web page. Depending on the nature of the web content the nuggetcould either be statically authored along with the page or it could bedynamically authored based on the dynamic content on the web page.

For web sites that do not support Nuggets as part of the interface,nuggets can be generated by a nugget generation service 519.

Referring to FIG. 3, the Nugget Generation Service 519 is activated whena special form of a M-Doc Reference 507 is passed to the M-Doc Retriever502 on the mobile device 100, which references a web page rather than anM-Doc. The Nugget Retrieval Service detects this and passes the requestonto the Nugget Creation Service 519.

The Nugget Creation Service 519 generates a nugget for the supplied webpage in one of two ways:

-   -   1. If the supplied web page comes from an explicitly supported        web application, it passes the web page onto the nugget        generator for that web application. Common web applications such        as: Google, e-bay, Yahoo, Wikipedia and Amazon could be        supported.    -   2. If the web page is not from a supported application, then the        contents is scaled to fit into an M-Doc 500.

The nugget generator for supported applications include knowledge of thestructure of the web page and the main purpose of the web application,and thus be able to extract information from the key fields and presentthat in a nugget. For example:

-   -   A Google web page could be distilled to a nugget by showing the:        search criteria; the top ten hits; how many other hits where        returned.    -   An Amazon web page could be distilled to showing the contents of        the shopping cart.    -   A Wikipedia web page could be distilled to the term and the        definition.        1.2 Applications

This section discusses some applications of Internet Based M-Documentprinting architecture. Any M-Doc 500 can be Netpage enabled.

1.2.1 Daily Subscription Services

Many people buy the daily newspapers to access a few small sections ofthe paper, eg. puzzles, crosswords or cartoons. Using M-Doc printing itis possible for a user to browse online to the content and then requesta printout of the content of their choice. To avoid having to browseeach day, a subscription service that “pushes” the user's desiredsections out to them each day can be set up. The service could use: SMS,MMS or e-mail to “push” the M-Doc References out to the mobile device.The user can then print then content of the subscription when required.

Some of the sections of a newspaper that would suit this form ofdistribution are:

-   -   Crossword puzzles: Crossword grid on one sheet; clues on        another.    -   Number puzzles, eg. Sudoku    -   Jokes    -   Cartoons    -   Local Weather        1.2.2 Navigation and Location Based Services

Web services such as “WhereIs” (www.whereis.com) provide both theability to get a list of directions to go from one location to anotherand/or a map. Using a mobile device's browser 504 the user can enter thedestination and their current location and then have the map anddirections delivered as an M-Doc 500 to be printed.

This gives a more convenient presentation of the map and directions torefer to while driving. Also in many regions in the world it is illegalto look at a mobile phone while driving, but it is not illegal toconsult a map or written directions.

Mobile devices 100 that support location services are able to supplytheir location automatically. In this case it is only necessary tospecify the destination to receive a map and/or a set of directions toprint.

As well as assisting navigation, location based services can be used topresent of list of possible destinations. For example:

-   -   A service that prints out the list of restaurants within walking        distance;    -   Directions to the closest Service Station (or any type of shop);    -   Directions to the closest public transport stop.        1.2.3 Company Business Cards

Corporate websites often promote a company's public image, this can beextended by providing the ability to print a “Company Business Card”that gives the general information about the company and its generalcontact details. As well as a general “Company Business Card”,individual departments could easily have their own business cards, eg.the Service Department contact details.

Web sites for companies normally have a page dedicated to how to findthem, that is, directions on how to get to their buildings from majortransport hubs. These sites could easily include M-Documents 500 showingmaps of:

-   -   How to get to the company's premises    -   Where the closest parking is    -   Where the closest Hotel is    -   Directions on how to navigate from one building to another.        1.2.4 Discount Coupon/Voucher

Discount coupons can be delivered by M-Documents 500. These can bedelivered via the web as part of an advertisement, either directly withthe advertisement containing a link to a M-Document 500 or a couponcould be delivered to the mobile device as a reward for clicking throughan advertisement to the companies web site.

A company could “push” out via SMS, MMS or e-mail vouchers to members oftheir loyalty scheme or just to the general public as a promotion.Another variation on this scheme is the ability to deliver a voucher orcoupon to a user who enters a competition or votes on-line. For example,using an SMS to vote on a reality TV show could result in an MMS beingreturned with a coupon for a prize, or voting from a web site couldreturn an M-Doc 500 with an advertisement and the possibility of aprize.

1.2.5 On-Line Receipts

When performing on-line transactions from a mobile device 100 a receiptfor the transaction can be returned via an M-Doc 500. This gives aprintout that can be filed with a user's other receipts. The receiptscan contain bar codes and/or be Netpage enabled to allow the transactionto be recalled on-line on demand. Some example on-line transactions thiscould be used for are:

-   -   Betting, the nugget can record: the selected options, the odds,        the money wagered and the possible payouts;    -   Banking, the nugget can be similar to an EFTPOS receipt;    -   Purchasing, the nugget can be similar to a shop receipt;    -   Paying bills;    -   Taxi payment        1.2.6 Ticketing

For tickets that do not require magnetic stripes it is possible todeliver them over-the-air at the time of purchase. This could included:Public transport tickets; Theme park ride tickets; Theatre tickets;Cinema tickets.

1.2.7 Web CAM Print

While viewing a web cam on your mobile device, a user can select printand have an M-Document 500 of the image at that time sent to the usersphone.

1.2.8 On-Line Gaming

On-line games can use Nuggets to provide additional information aboutthe game. They can be used to provide: Cheat sheets; Maps; Charactersummaries; Brag cards, to demonstrate what level you have reached;Vouchers or Coupons as rewards for achievement.

2. Player Architecture

A Netpage Player 520 is a physical or virtual device capable of“playing” requests of various types. A play request 521 consists ofthree parts:

-   -   1. The target 522, which specifies which player 520 the request        521 should be played.    -   2. The operation 523, which specifies the action to be        performed.    -   3. A set of values 524, which are supplied as parameters to the        operation 523′.

Typically play requests 521 arise in response to user actions. Forexample, the user clicks on a tagged surface with a Netpage pointer, orinteracts with an application that is in contact with the Netpagesystem. Play requests 521 can be used to provide a simple feedbackmechanism (such as a request to display a text string to the user), ormay be used to cause more sophisticated interactions with physicaldevices (such as setting the thermostat temperature on a home airconditioning system).

Individual players 520 can be associated with a user 525 as shown inFIG. 4. A user 525 may be associated with multiple players 520 each ofwhich supports the playing of possibly overlapping sets of PlayRequests521.

It is likely that in many cases a single physical NetpagePlayer device520 is shared between multiple users 525. For example, a hi-fi audiosystem in a family room may be configured as an audio player formultiple members of the family. For the sake of brevity, this sectionfocuses on cases where physical players are exclusively used by a singleuser, however, it will be appreciated that this section may be appliedto multiple users.

Central to the NetpagePlayer 520 concept is the notion of a PlayRequest521 which are objects that represent a request to perform an operationon some device. This sections describes various details related toPlayRequests 521.

2.1 Structure of a PlayRequest

A PlayRequest (see FIG. 5) consists of three parts, each of which isoptional:

1. An optional target 522 which specifies which Netpage player 520 therequest 521 should be played,

2. An optional operation 523 which specifies the type of action to beperformed on the target player 522, and

3. An optional list of values 524 which are supplied as parameters tothe operation 523.

A play request 521 may either be fully or partially specified. A fullyspecified play request completely specifies all of the information(target, operation, and parameters) required to unambiguously deliverthe request to the target 522 and to perform the desired operation 532.A partially specified play request provides some indication of therequest 521 to be played, but does not provide enough information inorder for the play request 521 to be successfully delivered and playedwithout further processing.

A target 522 may either fully or partially specify the target 522 of aplay request 521. A fully specified target completely identifies thephysical player 520 on which the request 521 should be played. Apartially specified target provides some indication of the desiredtarget 522, but does not provide enough information in order for theplay request 521 to be delivered without further processing. Anoperation 523 can also be fully or partially specified. A Value 524consists of a physical type 525 and associated data 526. For example,the physical type 525 might be “image/jpeg” and the data 526 would bethe binary image data.

2.2 Values and Types

A Value 524 represents an instance of some physical type 525. Each Value524 has an associated physical type 525 and zero or more associated typecategories. The physical type identifies the structure of the dataelement of the Value 524. A possible mechanism would be to use MIMEtypes. For example, if the physical type 525 is image/jpeg then the dataelement would contain the binary data of an image in jpeg format.

Referring to FIG. 6, a Value is also optionally associated with one ormore Categories 527. A Category 527 is used to provide additionalinformation about the value 524 which may allow it to be handled moresensibly by the system (i.e. to allow a PlayRequest to be better matchedagainst the capabilities of candidate targets during request routing).As an example, an image value produced by a digital camera may have thephysical type 525 image/jpeg, but may also be associated with a category527 of “photo”, whereas an image value produced by a fax package mightalso have the physical type 525 image/jpeg, but could be associated witha category 527 of “facsimile” or with no category at all.

RequestRouters can take into account both the physical type 525 of avalue 524 and the categories 527 to which it belongs when determiningthe most appropriate way to handle a request 521.

2.3 Sample PlayRequests

To better demonstrate the PlayRequest concept, this section provides anumber of sample PlayRequests 521. A PlayRequest 521 can be viewed intabular form as shown below by example in Table 1.

TABLE 1 target <identification of the target of the request> operation<the name of the operation to be performed> parameters <physical typeand categories of parameter 1> <value of parameter 1> . . . . . .<physical type and categories of parameter n> <value of parameter n>

Firstly, the Request 1 shows a fully specified PlayRequest 521 fordialing a number on a specific mobile phone.

Request 1 target mobile-phone-xyz56474238 operation dial parametersphone-number “555 6754”

The target field 522 is fully specified and indicates that the request521 is to be performed on the device 520 identified by the name/address“mobile-phone-xyz56474238”. Note that for simplicity, simple textstrings to indicate the address of each physical target 522. The playrequest 521 contains an operation of “dial” which is understood by themobile phone's NetpagePlayer 520. The request 521 also includes a phonenumber which is a required parameter 528 to the “dial” operation 523.

As shown below, Request 2 is only partially specified due to onlycontaining a partially specified target 522. The target 522 specifiesthat the request 521 should be played on a mobile phone, but does notspecify which mobile phone.

Request 2 target mobile phone operation dial parameters phone-number“555 6754”

Request 3 shown below is also only partially specified. In this case,the target 522 has been completely left out. Although only beingpartially specified, the request 521 has a definite meaning: “dial thephone number 555 6754”. The device to be used to dial the number isstill to be determined.

Request 3 target operation dial parameters phone-number “555 6754”

Request 4 is even less fully specified that Request 3. Request 4 simplycontains the phone number “555 6754”. The operation to be performed withthe number and the device to handle the request (the target) is still tobe determined and the device to handle the request (the target) is stillto be determined.

Request 4 target operation parameters phone-number “555 6754”

Request 5 contains a fully specified target 522, but does not specify anoperation 523. Thus, the target of the request 521 is known, but whatthe target 522 is to do with the request (the operation) is still to bedetermined.

Request 5 target mobile-phone-xyz56474238 operation parametersphone-number “555 6754”

Request 6 can be used to send a simple text message to the user. Thetarget 522 is not specified, so the request means display the followingmessage on whichever player is the most appropriate at the current time.

Request 6 target operation display parameters text “Temperature inSydney is 28° C.”2.4 Invocation of PlayRequests

Play requests 521 can arise in one of two ways:

-   -   1. The user interacts with a printed Netpage form that has been        authored to include invocations of play requests 521.    -   2. An arbitrary application sends a play request 521 to a        Netpage Server 529.

These are discussed in the following sections.

2.4.1 Authored PlayRequests

PlayRequests 521 can be authored directly into a printed Netpagedocument. FIG. 7 provides an example of an interactive business card530. The business card 530 contains interactive elements 531 that can betriggered by clicking on them with a Netpage pointer. Each interactiveelement causes a PlayRequest 521 to be invoked. The Netpage Server 529then arranges for the request 521 to be played by routing it to theappropriate player device 520.

For example, referring to FIG. 7, consider a user with a Netpage-enabledmobile phone device 100 with a built-in Netpage pointer. The user clickson the mobile phone icon 532 on the business card 530 which causes thePlayRequest 521 to be triggered. The server 529 routes the play request521 to the user's mobile phone 100 and upon receiving the request 521,the mobile phone 100 commences dialing the required number.

2.4.2 Application Invoked PlayRequests

FIG. 8 shows a typical example of an application invoked PlayRequest521. The steps are as follows:

-   -   1. A Netpage pen 533 transmits a digital ink stroke 534 to the        Netpage Server 529.    -   2. The stroke 534 is determined to be a request 521 to submit a        Netpage form for processing.    -   3. The form is submitted to the corresponding Application 535.    -   4. As part of the form submission processing, the application        535 requests that a play request 521 be played by a Netpage        Player 520 associated with the user 525 who made the submission.    -   5. The Server 529 determines the target device 522 and relays        the play request 521 to that device 522.        2.5 Player Devices

Netpage Player instances can be deployed on various Player devices(platforms). Individual players support some subset of the full range ofPlayRequests 521 supported by Netpage. Table 2 shows some examples ofNetpage Player Devices.

TABLE 2 Example Netpage Player Devices Player Device Comments DesktopPlayer This device is a powerful computing unit usually with fixednetwork connectivity. A 585 desktop player is capable of playing a widerange of PlayRequests (e.g. audio, video, image, html, etc). The playercan interact with various external software/hardware components runningon the device. Thin Mobile This device is a mobile unit with limitedcomputing power such as web-enabled or Player low-end mobile phones. 587The thin player running on such mobile device is capable of playingvarious PlayRequests by utilizing the capabilities of the device.Examples include sending SMS objects, dialling phone number objects,etc. Due to the processing limitations of the device, a remote server(Netpage and Application Server) processes the input strokes/clicks andprovides the objects to the player. Smart Mobile This device is a mobileunit with more computing and storage capabilities, such as a Playerhigh-end smart mobile phone or a PDA. 588 Such device is capable ofenabling most of the Netpage functionality by running a Micro edition ofthe Netpage Server locally. In such an environment, the player canreceive PlayRequests from the local server (running on the device) andno on-line connectivity to a remote server peer would be necessarilyrequired at the time of playing. Embedded An embedded player is a customdevice that is built for a specific application. Player Examples of suchplayers are Digital Camera, capable of playing (i.e. showing) images andpossibly video; or Audio Player, capable of playing audio. The Netpageplayer is either built into the device or as a detachable unit. Notethat while some of the devices mentioned in Table 2 are also capable ofrecording/capturing objects (e.g. a digital camera of a mobile phone iscapable of capturing images), access to such captured data is notaccessible via the Netpage Player concept, but can be accessed via aNetpage Clipboard, which will be discussed in more detail.

Note that while some of the devices mentioned in Table 2 are alsocapable of recording/capturing objects (e.g. a digital camera of amobile phone is capable of capturing images), access to such captureddata is not accessible via the Netpage Player concept, but can beaccessed via a Netpage Clipboard, which will be discussed in moredetail.

2.6 Request Processing

This sections describes how PlayRequest objects 521 are processedthroughout the Netpage system. The processing of requests includes twooperations:

-   -   1. The routing of requests 521 from one NetpagePlayer 520 to        another.    -   2. The transformation of PlayRequests 521 (for example to change        a partially specified request more specified) as they are being        routed.        2.6.1 Request Routing

A play request 521 is routed from source to the eventual destination viaone or more intermediary RequestRouters 536 as shown in FIG. 9.RequestRouters 536 implement the NetpagePlayer interface, and areresponsible for routing each PlayRequest 521 they receive to anappropriate target NetpagePlayer 520. Each RequestRouter 536 maintains aset of potential targets 538. Eventually a PlayRequest 521 arrives at aPlayerAgent 537 which is responsible for actually performing the playrequest 521.

All PlayRequests 521 from (or on behalf of) a user 525 are initiallyhandled by a RequestRouter 536 inside the Netpage Server 529. Thisrouter is called the UserRequestRouter 539 (see FIG. 10). Typically theUserRequestRouter 539 forwards requests to a RequestRouter 536 residingon a physical device, although such forwarding may pass through anarbitrary number of intermediary RequestRouters 536 along the way.Device based RequestRouters 536 are responsible for routing requests 521to the various player agents 537 running on the device.

The typical scenario is shown in FIG. 11. The UserRequestRouter 539 hasa set of potential targets 538 which are all RequestRouters 536 residingon physical devices. Each device's RequestRouter 536 then has a set ofpotential PlayerAgent targets. The PlayerAgents are the NetpagePlayers520 that actually play requests 521.

2.6.2 Request Transformation

Each RequestRouter 536 can optionally transform the PlayRequest 521 itreceives before passing it on to a subsequent NetpagePlayer 520. Thetransformation typically produces a more fully specified version of thesupplied PlayRequest 521, but may also produce a completely newPlayRequest 521 with no fields in common with the source PlayRequest521.

2.6.3 Player Capabilities

Different players have different capabilities. That is, each player iscapable of playing a different set of PlayRequests 521. The capabilitiesof a NetpagePlayer 520 are specified in a CapabilitySpecification 540.The capabilities 541 of each child player are taken into account byRequestRouters 536 when handling PlayRequests 521. The capabilities 541of different players may overlap, potentially resulting in ambiguousPlayRequests 521. Such ambiguities are resolved by RequestRouters usingmethods described in further detail below.

The CapabilitySpecification 540 is not limited to simply specifyingwhich operations 523 can be performed on which value types 524. It mayalso specify finer grained details. For a specific PlayRequest 521 theCapabilitySpecification 540 might specify that it can only handle asubset of possible values. For example, a player 520 that supports theplaying of audio objects could place a limitation on the size of audioobjects supported.

2.6.3.1 Capability and Request Propagation

As shown in FIG. 12, a PlayerAgent 537 advertises its capabilities 541to its parent RequestRouter 542 which in turn propagates the aggregationof its children's capabilities 544 to its parent RequestRouter 543.Eventually the propagation reaches the UserRequestRouter 539 which thenhas an overall view of the capabilities 545 of all of the players at itsdisposal. PlayRequest propagation moves in the opposite direction.Requests 521 start at the UserRequestRouter which determines the mostappropriate child to which the request should be sent. The (possiblytransformed) request is sent to the selected child which in turnpropagates the request to one of its children. Eventually the requestreaches a PlayerAgent 537 which is responsible for actually playing therequest.

2.6.3.2 Capability Aggregation and Transformation

As a RequestRouter propagates player capabilities (541, 544, 545) to itsparent RequestRouter 536, it may perform capability aggregation andtransformation. Capability Aggregation is where the router 536 combinesthe capabilities of its children into a single capability specification548. Capability Transformation is where the router 536 modifies theadvertised capabilities of its children due to capabilities (or perhapslimitations) of the router itself.

FIG. 13 provides an example of a simple capability aggregation. Therouter has two children 550, 551, the first child 550 of whichadvertises the capability to display jpeg images, the second childadvertises the capability 553 to display plain text. The router 536 thenaggregates the child capabilities 552, 553 into a single capabilityspecification 540 which is capable of displaying both jpeg images andplain text.

FIG. 14 provides an example of a capability transformation. ThePlayerAgent 537 advertises its capability 555 to display image/jpegs.The RequestRouter 536 has access to an image converter 559 that canconvert images in png format to jpeg format. As such, the capabilityspecification 555 is transformed before propagation into a capabilityspecification 556 that includes the ability to display files in pngformat as well as in jpeg format.

2.6.4 Dynamic Capabilities

The capabilities advertised by a particular NetpagePlayer 520 can changeover time. For example:

-   -   1. Additional hardware or software can be installed/removed        to/from a device, enabling the player to support more/less        PlayRequests 521.    -   2. The maximum object size supported by a player may change        depending on the spare capacity in the player's memory.

3. A mobile player might be capable of receiving streaming media when itis connected to the network through a high-bandwidth network.

-   -   4. Common user interactions with the player (e.g. starting an        application, changing a setting) can cause the player to        advertise more or less capabilities.

Such changes in capabilities are to be communicated to the player'sparent RequestRouter 536, and potentially, but not always, to theparent's parent, and so on all the way to the UserRequestRouter 539.

At the same time, as dynamic capability changes are being propagated,requests 521 are being routed in the opposite direction (as shown inFIG. 12). This creates a race condition between capability propagationand request routing, such that by the time a request arrives at aRequestRouter 536, either the request can no longer be handled by theplayer 520, or the player 520 is no longer the most appropriaterecipient for that request 521. Either case may require that a request521 be rejected by the player 520 (be it a router or an agent) andre-routed to the appropriate player 520.

Additionally, dynamic propagation of capability changes couldpotentially cause an undesirable level of network traffic, harmingoverall system performance.

2.6.5 Request handling by the UserRequestRouter

As already discussed, all PlayRequests 521 presented to theNetpageServer 529 are handled by the UserRequestRouter 539. The purposeof the UserRequestRouter 539 is twofold: (i) To determine the mostappropriate child NetpagePlayer (the target 522) to which the request521 should be routed; and (ii) To determine any required transformationsto the request 521 that are necessary in order for the selected target521 to be able to handle the request 521.

In order to determine both of the above, the UserRequestRouter 539 takesinto account the content of the PlayRequest 521 and the context withinwhich it is handled. The context includes a large range of factors,including, but not limited to the following:

-   -   1. The capabilities of each of the available children        NetpagePlayers. Availability being partially determined by the        user identity.    -   2. The current contents of the Netpage clipboard.    -   3. The originating source of the request (e.g. the Netpage        pointer device which triggered the play request) and/or the        route via which the request arrived.    -   4. The current player profile.    -   5. The current date and time.        2.6.6 Player Profiles

It is possible that multiple players registered with a user support thesame PlayRequests 521. As a concrete example, consider the followingscenario where a user has registered the following players. The user 525has three registered players 520 all of which are capable of playingimages:

-   -   A camera phone for playing phone numbers, plain text, and        images.    -   A digital camera for playing images.    -   A desktop application for playing plain text, html, images,        video and audio.

Now consider the case where the UserRequestRouter 539 receives thefollowing partially specified PlayRequest 521:

target operation display parameters image contents of image

The request 521 could potentially be played on any of the devices 520mentioned. As such, the request 521 is ambiguous.

PlayerProfiles 557 are one mechanism which can be used in order to allowthe UserRequestRouter 539 to resolve such ambiguities. A PlayerProfile557 provides a (typically restricted) view of the set of players 520available for a particular user 525 and the set of PlayRequests 521 thatcan be played on those players 520. A user 525 may ha multiple playerprofiles 525 indicating the various scenarios within which they use theNetpage system. At any point in time, one of these profiles 525 is setas the Current Profile 558 as shown in FIG. 15.

For example, a user might have the following profiles:

-   -   An “office” profile that directs most player requests to their        desktop PC.    -   A “home” profile that directs requests to various devices        throughout the user's house.    -   A “mobile” profile that directs player requests to various        portable devices (e.g. a smart phone).    -   A “car” profile that directs player requests to devices within        the user's automobile.

A user can quickly change their current profile by a simple user action.For example, if in the office, the user could select the profile via adesktop GUI. Alternatively the user could use their Netpage pen/pointerto select a profile from a printed interface 559 such as that shown inFIG. 16. The system could also allow a user to specify regular scheduledtimes at which their current profile should switch.

2.6.7 An Example Request Routing

Consider the interactive business card 530 shown in FIG. 7. The businesscard contains a number of interactive elements including the two fields560, 561 highlighted in FIG. 17. Each of the fields 560, 561 isrepresented by a partially specified PlayRequest 562, 563. Clicking oneither field 560, 561 with a Netpage pointer causes the correspondingplay request 562, 563 to be submitted to the Netpage Server 529 forprocessing.

FIG. 18 provides an example of how such a field might be routed. At step564 we have the original play request as submitted to the server. Atstep 565 the UserRequestRouter interrogates the PlayRequest and thecurrent context and determines that the request should be routed to theuser's mobile phone. As such it transforms the original PlayRequest toproduce the request shown in step 566 and routes the play request to theRequestRouter on the mobile phone. At step 567 the mobile phone'sRequestRouter checks whether the phone is in a quiescent state (i.e. noapplications running). If so, it transforms the PlayRequest by settingthe operation to “dial”, and is routed to the voice communicationssub-system agent 568. If, however, the SMS creation application isrunning, then the PlayRequest is transformed by setting the operation to“set-destination-number”, and it is routed to the running SMS creationapplication 569. Lastly, if instead the Contacts application is running,then the PlayRequest is transformed by setting the operation to“add-contact”, and it is routed to the Contacts application 570.

The end result is that the act of clicking on a telephone number on thebusiness card can have very different results depending on the contextwithin which it is applied. In this example, both the current contextwithin the Netpage Server 529 and the current context on the mobilephone were taken into account when processing the PlayRequest 521.

2.7 Communicating with Players

Various communication methods are used between a Netpage Server 529,intermediate gateways 570 and the Netpage client to enable the playingof PlayRequests 521 on the Player device 520. Environmental factors suchas the following affect the selected communication mechanism:

-   -   Available network connectivity.    -   Type of player device being targeted.    -   Size and type of objects being transferred and the nature of the        operation being played.

This section describes categories of messaging mechanisms. Note that ina single play scenario a combination of messaging methods can be used.

2.7.1 RPC (Synchronous) Messaging

Referring to FIG. 19, in an environment where a persistent connectioncan be maintained between the Netpage Server 529 and the Player 520, theserver 529 can send the play request 521 to the player, block till playa request is handled and a response 571 is returned. A Desktop Player inan active session can communicate to the server 529 using this method.

2.7.2 Notification (Asynchronous) Messaging

Referring to FIG. 20, notification messaging is used when theenvironment allows playing of an Object through an asynchronous play( )request 521 initiated from the Netpage Server 529 to the Netpage Player520, delivering the object to be played. The server 529 may continue itsactivities and optionally receive a future response from the Player. Theexample of a notification based request delivery is when Netpage serverpushes an image to a Netpage Player.

Depending on the underlying network infrastructure, a suitable protocolis used to push notifications to the Netpage Player, i.e. WAP push, SMS,etc.

2.7.3 Streaming

Referring to FIG. 21, for certain media types, it is preferable to beable to stream data to a player rather than transmitting the entireobject before playing commences. The reasons are that:

-   -   Large objects may take significant time to transmit in their        entirety to the player. Streaming allows for playing to take        place before the entire object has arrived at the player thereby        reducing latency.    -   Target player devices may not have the capacity to hold the        entire object. In that case, streaming is one option for playing        the object on the device due to this limitation.    -   Unbounded objects such as live video can be transmitted by        streaming.

Video and audio provide the most significant examples of types that aretypically better suited to streaming. To enable the streaming NetpageServer would be involved in the player selection process and would thenleave actual streaming up to the two parties. Otherwise the server islikely to be a bottleneck and a source of additional latency.

Player can perform read-ahead operations to buffer the data ahead ofplaying and avoid network delays and jitters which can affect the user'sexperience.

2.7.4 Interactive Messaging

Referring to FIG. 22, in some scenarios multiple user/pointerinteractions with the Netpage Server 529 invoking multiple play requestsis required to complete a user play experience. For instance consider ascenario where a user has a Netpage printout photo that he/she wouldlike to send as a MMS message to a friend. One way of achieving this isby the user clicking on the friend's business card's MMS hyperlink. Thisaction sends a play request to the player activating the MMS editor withthe phone number to which the message is being sent. At this point usercan choose to click on the photo to attach it to the MMS message. Thisresults in the photo being sent as a second play request to the Player,wherein the photo is attached to the MMS content. The state of theplayer 520 allows chaining multiple play requests 521 to complete atransaction.

2.7.5 Hybrid Messaging

Referring to FIG. 23, in some scenarios a multi-transaction messagingwithout user interaction is performed to play an object. Some examplesof hybrid messaging are:

-   -   Consider a scenario where a low-end mobile phone player without        support for the suitable push-based notification, wishes to play        an unbounded object. A hybrid solution can be adopted to push a        small notification to the device, notifying the player        application (i.e. through SMS) to initiate a stream-based        communication to the Netpage Server request (i.e. WSP) for        delivery of the object.    -   Displaying a URL object also uses a multi-transaction hybrid        messaging, where the original object (the URI) is pushed to the        device using a Notification message. At this point the player        (without user interaction) retrieves the URI content by sending        a synchronous request/response message through HTTP.        2.7.6 Player Session Establishment

When players 520 are instantiated on devices (for instance during userlogin on a desktop player or on-demand by the mobile user), the player520 registers the mobile device 100 is available for play requests byinitiating a player session 580 with the Netpage Server 529.

FIG. 24 shows the basic lifecycle of a NetpagePlayer session 580. First,a process, which is typically running on a remote machine/device, callsthe startPlayerSession( ) method 581 to commence a NetpagePlayer session581 with the Netpage Server 529. A userId 582 is provided whichindicates the user 525 to which the supplied player 520 should beassociated. Upon reception of a startPlayerSession( ) request 581, theserver 529 creates a PlayerSession object which is returned to theremote process 583. This object can be used at some later time toterminate the session by calling the terminate( ) method.

FIG. 25 shows more details of the handling a player session 580 withinthe server 529. As shown previously, a player session is created 589 bycalling the startPlayerSession( ) method on the Netpage Server'sCommandProcessor. This causes the creation of a PlayerSession which inturn creates a NetpagePlayerProxy which is run within the server andacts as a proxy for the real player by implementing the NetpagePlayerinterface and passing all requests on to the real player.NetpagePlayerManager::addSession( ) is called to register the sessionwith the NetpagePlayerManager. The NetpagePlayerManager is a singletonobject responsible for managing all NetpagePlayer sessions and also forcoordinating all NetpagePlayer traffic within the server.

2.8 Player Deployment

2.8.1 Player Connectivity

Netpage Player 520 can be deployed on various devices with differentnetwork connectivity capabilities. Table 3 lists some examples of playernetwork connectivity and transmission mechanisms. Note that a deploymentenvironment may utilize a combination of connectivity types. Eachconnectivity type explains how one hop communicates to the next hop. Forinstance a player communicates with a wireless gateway on the path tothe Netpage Server.

TABLE 3 Netpage Player Connectivity Types Player Connectivity CommentsExample Standards Local On a desktop device or a smart mobile, theplayer communicates Shared memory, with the next hop locally through theon-board IPC mechanisms. TCP/IP on the loop- The next hop could be theserver running on the same desktop, or back interface, etc. the microserver running on the smart mobile, or the gateway to the remote serverrunning locally. Fixed Network The connectivity to the next hop is overa fixed (wired) network. TCP over the Internet For instance the desktopplayer communicates to the remote through a dial-up serial serverthrough IP-based protocols over Internet. link, etc. Short-range Theplayer connects to the next hop through a Wireless Personal IrDA,Bluetooth, wireless Area network (WPAN). For instance a player runningon a PDA 802.15, etc. uses Bluetooth to communicate to a Relay modulerunning on a local desktop. Long-range The player communicates with thenext hop through a long-range 802.11a, b, g, GPRS, wireless wirelessnetwork with national (WLAN) or global (WWAN and WCDMA, etc. Satellite)coverage. For instance the player is a WAP-enabled cell phone connectingto the remote server over GPRS wireless network.2.8.2 Player Deployment Configuration

Different combinations of player devices and connectivity types can beconfigured to provide a suitable Netpage Player and Server integrationenvironment. FIGS. 26 to 29 demonstrate some of the more widelyapplicable types of deployment environments. Example pen connectivity isdemonstrated for clarity.

3. Object Association Design

3.1 Object Associations Overview

A Netpage object association allows content (objects 601) to beassociated with locations 513 on Netpage documents 603 and printouts.Arbitrary content types can be supported, but common examples includepictures/photos, audio, and video.

An object 601 can either be associated 616 with a document during theauthoring of the document (static association) or can be associated witha particular document printout at some arbitrary time after thatprintout has been created (dynamic association).

Additionally, an association 616 may either be an impression associationor a field association. An impression association is a dynamicassociation that is associated with a particular {x,y} location 613 onan impression. A field association is associated with a particular fieldof a form (static association) or form instance (dynamic association).

3.2 Communicating Status to the User

At times the mechanisms described require communication of statusinformation (often errors) to the user. The simplest way to achieve thisis to make use of the Netpage Player infrastructure. Special operations(e.g. show-status-ok-message and show-status-error-message) can bedesignated for transmitting status information to the user. The playerarchitecture would determine, for each message, the most appropriatedevice (or devices) on which to display the message and the way in whichto display it. For example, it may be that in certain situations thepointer is the only available player, in which case an error statusmight be “played” by illuminating a red LED on the pointer or playing ashort sound.

3.3 Impression Associations

An object 601 can be associated dynamically with a specific {x,y}location 613 on an impression. The associations 616 are stored insidethe Netpage Server 529 and can then be viewed in a number of ways:

-   -   1. By interacting with the physical impression using a device        that communicates with the Netpage Server 529 to retrieve the        associated objects 601.    -   2. By interacting with a virtual view of the impression from        within a graphical software application 602.        3.4 Interacting with Dynamic Associations

Referring to FIG. 30, an associated object 601 can be viewed byinteracting with the physical impression using a Netpage Tag awaredevice. This shows an example where a tag reading device with built inscreen is passed over a physical impression. The device communicatesimpression locations to the Netpage server 529 which responds withinformation regarding any associated object 601 that is currently sensedby the device. Depending on configuration, the device might beginimmediately playing each associated object or present the user 525 withthe option of playing the object 601. If multiple objects 601 are withinthe device's field of view, then the device could present the user witha list of the objects 601 for selection.

Associated objects 601 can also be viewed by interacting with a virtualview of the impression from within a graphical software application 602.There are a number of possible mechanisms for displaying associatedobjects 601 in such an application 602. Each mechanism is responsiblefor:

-   -   Indicating the presence of an associated object (or objects        601), and    -   Displaying the actual content of an associated object 601.

An example mechanism is shown in FIG. 31 where a graphical “NetpageExplorer” application 602 is being used to view an impression whichcontains associated objects 601. The example shows an approach in whichsome visible token 603 (in this case a black star) is displayed thatindicates the locations that contain object associations 616. Thedisplay of the tokens 603 can be toggled by clicking on a toolbar button609. To view an associated object 601 the user clicks (or alternativelydouble-clicks) on the relevant token 603 which causes the NetpageExplorer application 602 to retrieve the associated object 601 from theNetpage Server 529 and to then play the object 601. Depending on thecapabilities of the Netpage Explorer application 602, the object 601 caneither be played within the Netpage Explorer application 602 itself, orby an external application.

3.5 Methods for Creating Impression Associations

This section describes various alternative techniques that may beprovided to allow users to dynamically associate content with a locationon an impression. They include:

-   -   Modal association mechanisms    -   Sticker based mechanisms    -   Swipe Based mechanisms    -   Swipe Based Stickers        3.5.1 Modal Association

A Modal Assocation involves first placing the user/pen session in a modein which the next pointer click is interpreted as the specification ofan impression location to which a current clipboard object 601 should beassociated.

FIG. 32 shows an example of how an impression object association 617might be created modally. In this case, the user attaches a photo to animpression. The steps are as follows:

-   -   1. The user takes a photograph using their digital camera 610.    -   2. The user pushes the photograph to the Netpage clipboard on        the Netpage Server 529 (possibly implicitly).    -   3. The user clicks with their Netpage pointer 533 on a printed        toolbar 611. In this case, the user clicks on the “Attach        Object” button 612. This places the pointer session into a mode        in which the next click with the pointer 533 is interpreted as        the specification of an impression location to which the current        clipboard object 601 should be associated.    -   4. The user clicks on the desired location 613 on a printed page        614. This causes the Netpage Server 527 to retrieve the current        object 601 from the user's clipboard 615 and to associate the        object with the impression location selected in step 3.

An alternative approach requires steps 3 and 4 to be performed in theopposite order. That is, the “Attach Object” command is interpreted tomean associate the object with the impression location most recentlytouched by the user.

Modal association mechanisms can be implemented on top of the NetpageClipboard 615 mechanism.

3.5.2 Tagged Stickers

A tagged sticker 620 is a physical adhesive sticker which is Netpage tagencoded 617. That is, it is a Netpage impression printed onto a physicalsticker. Clicking on a tagged sticker causes an object to be associatedwith that sticker (impression). Tagged stickers 620 can be physicallyattached to any surface whether it be tagged or otherwise (eg. books,desks, walls, etc) and thus provide a very flexible mechanism fordynamically associating objects with locations.

FIG. 33 shows an example of a simple tagged sticker 620. In order toassociate an object 601 with the sticker 620, the user would perform thefollowing steps.

-   -   1. Push object 601 into Netpage Clipboard 615    -   2. Physically paste sticker 620 onto any surface    -   3. Use Netpage pointer 533 to click on the sticker 620 to        associate the object 620.

Once an object 601 has been associated with a sticker 620, there arevarious ways in which the user can retrieve/play the object 601.Firstly, the object 601 can be interacted with using a physical NetpagePlayer device 520. Secondly, simply clicking on the sticker 620 with aNetpage pointer 533 would cause the object 601 to be played. This latterbehaviour suggests that sticker associations would actually beimplemented as field associations 618.

3.5.3 Reusable Stickers

As so far described, once an object 601 is associated with a sticker620, that association 616 cannot be altered. A reusable sticker 621allows for the object associated with a sticker to be changed, orerased. Such a sticker 621 is shown in FIG. 34. The “Attach” button 622is used to associate an object 601 with the sticker 621 and allows for anew object 601 to be associated with the sticker 621, overwriting anyprevious association 616. The “Clear” button 623 allows for anyassociation 616 to be removed.

Both “Attach” 622 and “Clear” 623 are destructive operations in thatthey remove any association 616 that may have been in place before theoperation took place. As such, it may be desirable to be able to protectagainst accidental invocation of such operations, especially in thesticker scenario in which the entire sticker is a clickable area.

One mechanism for doing that is shown in FIG. 35 in which a sticker 621has a “Confirm Action” button 624. In order for a destructive operationto be confirmed, the user must first select the operation andsubsequently select the “Confirm Action” button 624. A suitable timeout(say 10 seconds) can be used such that confirmations must take placewithin the timeout period in order to be valid.

An alternative for preventing accidental invocation of destructiveoperations is to limit the interactivity of the sticker to small areaswithin the sticker as shown in FIG. 36. The associated object 601 isonly played when the user selects the “Play” operation 625. The overallsticker 621 is not interactive. As such, accidental invocation ofdestructive operations should be much less likely.

3.5.4 Category Specific Stickers

So far we have described stickers which retrieve the object 601 mostrecently assigned to the Netpage Clipboard 615. The Netpage Clipboard615 can store multiple objects 601 per user 625 with each object 601falling into an object category or set of categories. As such, it ispossible to have Category Specific Stickers 627 that retrieve currentobjects from the Netpage Clipboard 615 by category.

FIG. 37 provides examples. Clicking “Attach” on the left sticker causesthe current clipboard object 601 with a category of “video” to beassociated with the sticker 620. The sticker on the right achieves asimilar effect for objects in the “photo” category.

3.5.5 Swipe Based Mechanisms

The printed toolbar in FIG. 38 allows an impression association 617 tobe created by swiping a command from a printed toolbar 626 to thelocation on an impression to which the object 601 is to be associated.The user temporarily places the toolbar 626 on top of the destinationimpression and swipes from the toolbar 626 to the impression. Theswiping action provides the system with digital ink samples from boththe toolbar and the destination impression. These samples enable thedetermination of both which object 601 is to be associated (for the cardshown the possibilities being the current video clip, current photo,current audio clip, or in the case of the “Any” icon, the current objectregardless of type) and to which impression and location on thatimpression the object 601 is to be associated.

3.5.6 Swipe Based Stickers

A limitation of the stickers described earlier is that they do notactually create an association 616 between the object 601 and theunderlying impression on which the sticker is applied. For example,consider the situation where a sticker has been applied to a taggedimpression, and subsequently an object 601 is associated with thesticker. If the underlying impression is viewed within the NetpageExplorer application, then the sticker and associated object 601 is notbe displayed since the Netpage Server 529 is not aware that the stickerhas been applied to that impression.

The above problem can be solved by applying the swipe based approach tostickers. The user steps involved are:

-   -   1. Push object 601 into Netpage Clipboard 615    -   2. Physically paste sticker 620 onto a tagged impression 600    -   3. Use Netpage pointer 533 to swipe from sticker 620 to        impression 600.

The action of swiping across both the sticker 620 and the impression 600creates a triple association between the impression 600, the sticker620, and the object 601. Specifically, the Netpage server 529 is nowaware of:

-   -   The sticker 620 to which the object 601 has been associated, and    -   The impression 600 on which the sticker 620 has been placed, and        the location 613 on the impression 600 at which the sticker 620        has been placed, and thereby the location 613 on the impression        600 to which the object 601 is associated.

The above associations 616 allow the object 601 and sticker 620 to bedisplayed inside tools such as Netpage Explorer 602.

A variant of the swipe based sticker is shown in FIG. 39. It includes atransparent region 628 hich allows the tags of the underlying impression600 to be seen through the sticker 620. This makes it possible to createa triple association 616 with a swipe that remains within the confinesof the sticker 620.

In addition, the transparent region 628 may be transparent in theinfrared spectrum in order for the Netpage pointer 533 to be able to seethe Netpage tags on the underlying impression 600. This allows for thetransparent region 628 to be non-transparent in the visible spectrum. Assuch, graphics can be printed over the transparent region 628. Forexample, a swipe based category specific sticker can be constructedwhich includes indicative graphics printed over the transparent region628 as shown in FIG. 40. It is also possible that the instead of using atransparent region 628, the sticker 620 may alternatively include ahole.

3.5.7 Impression Associations Object Model

FIG. 41 shows the basic object model representation of impressionassociations 617. An ImpressionAssociation 617 includes a location andthe object's content. An ImpressionLocation includes an {impression,x,y}tuple 629. The content of an associated object is represented as aPlayRequest 521. In the common case, the PlayRequest 521 includes asingle value, but it also possible to associate targets and operations.For example, any PlayRequest 521 can be associated with an impression.

A swipe based sticker is associated with an ImpressionAssociation on theunderlying impression.

3.6 Field Associations

An object 601 can be dynamically associated with a particular field of aform instance 630. Such an associated object 601 is then delivered tothe relevant application as part of a submission of the form instance630. This can be used, for example, to dynamically attach images (e.g.photos) to a Netpage form 632 and to then have those images sent to theapplication when the user clicks on the “submit” form command.

The relationship between field associations 618 and impressionassociations 617 is shown in FIG. 42. A FieldAssociation 618 consists ofa specification of the field with which the object 601 is associated anda reference to the underlying impression association 617 which providesdetails of the actual object 601. The ImpressionAssociation 617structure has been described. The field is specified by aFieldInstanceAddress 631 which specifies a form 632 instance 630 and afield number 633 within that form instance 630. A FormInstance 630 is aspecific instance of a Form 632 which is printed on to a Printout 633.

3.6.1 A Sample Application

FIG. 43 shows a simple application that demonstrates the use of fieldassociations 618. Use of the application proceeds as follows:

-   -   1. The user clicks on the “Click here to attach photo” button        640 with a Netpage pen 533 or pointing device. This causes the        application to request the current photo from the Netpage        clipboard 615 associated with the user 525. That photo is then        associated with the large rectangular field 641.    -   2. The user can click on the large field 641 in order to        retrieve the associated photo and display it on a suitable        Netpage Player device associated with that user 525.    -   3. The overall form can be submitted by clicking on the “Submit”        button 642. This causes a form submission to be sent to the        relevant application. The form submission includes the photo        currently associated with the large field 641.

The following sections use the sample application to describe theremaining details of the field association mechanism.

3.6.2 Form Design for the Sample Application

In order to design a form 632 that supports a simple application, twoforms are specified. The first form 643 is a standard application form.The second form 644 is a special form that include fields that share thesame impression coordinates as elements from the first form, but areoverlayed on top of the first form's fields.

The underlying (first) form 643 is shown in FIG. 44. This includes asubmit button 642 for the application and also a field of typeObjectReceiver 644. It is to this second field 644 that an object is tobe assigned. The actual assignment occurs as a result of userinteraction with the overlayed form as described below.

The overlayed form 644 is shown in FIG. 45. The application name for theoverlayed form 644 is the system supplied “sys:object-associator”application. This is a standard internal application that expects toreceive a form submission with a “submit” button having one of the namesas shown in Table 4. The table describes the meanings of each of thepossible submit buttons. Note that an object-associator form does notneed to provide all four submit buttons, although it must minimallyprovide both a “set” and a “target” button.

TABLE 4 Submit buttons supported by the sys: object-associatorapplication Name Description set The set command indicates that anobject of the specified type is to be fetched from the Netpage clipboardand associated with an ObjectReceiver field on the underlying form. Thelocation of the relevant ObjectReceiver field is indicated by the“target” field that must also be part of this form. The type of objectto be associated is indicated by the field's value. If it is left blank,then the current clipboard object (regardless of type) is retrieved andassociated. clear The clear command indicates that the objectassociation currently assigned to the “target” field should be removed.show The show command causes the object associated with the target field(if any) to be displayed on a suitable Netpage Player. target The targetfield has two roles. Firstly it indicates the location/field on whichthe set, clear and show commands should act. Secondly, if the targetfield is clicked then it behaves as if it is a show command.

The below pseudo code outlines the handling of the set command. The mainprinciple to note is that the center of the target field provides the{x,y} location to which the retrieved object should be associated. This{x,y} location is also used to locate a form field with which the objectis to be associated. This is achieved by determining the uppermost fieldon the page that intersects with {x,y} and that is also receptive toobject associations. In the case of the example, the uppermost field onthe page that intersects {x,y} is the target field itself, but it beinga submit field, is not receptive to object associations. The nextuppermost field that intersects {x,y} is the ObjectReceiver field on theunderlying form. The end result is that the object is associated with afield on the underlying application form rather than the overlayedsys:object-associator form.

ObjectAssociatorApp::handleSetCommand(a_submission, a_formDescription) { let l_setField = a_submission.getSubmitButton( )   // fetch the targetfield for the current form  let l_targetField =a_formDescription.fetchField(“target”)   // Calculate the impressionlocation to which the object is to be   // associated. The center of the“target” field is used as the   // location to place the object.  letl_targetCenter = l_targetField.center( )  let l_location =    ImpressionLocation(a_submission.getImpressionId( ),    targetCenter);   // Look for a receptive field to which we canassociate the object.   // A receptive field is a field with a type thatindicates it is   // receptive to object associations. “submit” fieldsare not   // receptive, but most other field types are. N.B. Thereceptive   // field is typically on another form.  let l_targetField =     (the uppermost receptive field on the page that includesl_location)   // Retrieve an object of the appropriate type from theNetpageClipboard.   // The type to retrieve is indicated by the “value”element of the   // “set” field  let l_objectRequestType =l_setField.value( )  let l_clipboard = getClipboard(l_submission.userId())  let l_object = l_clipboard.fetchObject(l_objectRequestType)   //create the associations  let l_impressionAssoc =createImpressionAssociation(l_location,  l_object) createFieldAssociation(l_targetField, l_impressionAssoc) }3.6.3 Field Associations and Form Submissions

Netpage forms 632 generally have one or more form submission buttons642. Clicking on such a button 642 with a pointer 533 causes the Netpageserver 529 to perform handwriting recognition of the digital ink (ifany) assigned to the form 632, and to bundle the result into a formsubmission which is then posted to the application associated with theform 632. The handwriting recognition largely involves convertinghandwriting fields (such as textual combs, and check boxes) into theirdigital equivalents (for example digital ink in textual combs isconverted to text).

Associated objects 601 can optionally be submitted as part of such formsubmissions. The requirement to transfer an associated object 601 (orotherwise) as part of the form submission is specified in the formdefinition for that form.

3.7 Static Associations (Embedded Objects)

Objects 601 can be associated with a document at document creation time.As with dynamic associations, such associations can either be impressionassociations 617 or field associations 618. Static associations arespecified in the Interface Description for the document. Staticassociations are represented as PlayRequests 521.

4. Netpage Clipboard

The Netpage Clipboard 615 is a system supplied, per user objectrepository to which the user can push an object 601. The object 601 thuspushed is considered to be the user's “current object” which may then beaccessed by applications, in particular by the UserRequestRouter, but ingeneral by any application that is acting on behalf of the user.

4.1 Representation of Clipboard Objects

Clipboard objects are stored as Netpage Player PlayRequest objects 521.A PlayRequest 521 corresponds to a request to perform an action. Itconsists of three parts:

-   -   1. A target, which specifies on which player (device) the        request should be executed.    -   2. The operation, which specifies the action to be taken.    -   3. A set of values, which are supplied as parameters to the        operation.

A Value represents an instance of some physical type and consists of atype specification and data. The type specification has a physical typeand zero or more associated type categories. The physical typeidentifies the structure of the data element of the Value. This documentdoes not specify a particular representation for physical types. Apossible mechanism would be to use MIME types. For example, if thephysical type is image/jpeg then the data element would contain thebinary data of an image in jpeg format.

A shorthand form of specifying PlayRequests will now be used in thissection. As an example, instead of using the tabular form this sectionwill use the following syntax for PlayRequests that only have a singleitem:

-   -   value {phone-number, “555 3473”}

target operation values phone-number “555 3473”4.2 Mechanisms for Pushing Values to the Clipboard

In general, a user performs an explicit user action in order to push avalue to the clipboard. Such actions may take various forms as describedin the following sections. Once a value is in the clipboard anyapplication can access the value. For example the value can bedynamically associated with a form field on a printed impression.

4.2.1 Push via Printed Netpage Form

Values can be pushed to the clipboard by interacting with a printedNetpage form such as that shown in FIG. 46. The form has been authoredsuch that clicking on any of the phone numbers causes that phone numberto be sent to the Netpage clipboard 615 for that user 525. For example,clicking on the phone number for “Susan Wilson” 615 causes the followingvalue to be pushed to the clipboard 615:

-   -   value {phone-number, “151 425 0617”}        4.2.2 Push via Physical Device

Values 524 can be pushed to the clipboard 615 by Netpage aware devicesthat are capable of capturing and/or storing typed objects. Table 5provides some example devices and scenarios in which they might pushvalues to the Netpage clipboard 615.

TABLE 5 Examples of Netpage aware devices capable of pushing objects tothe clipboard Device Example scenario Digital Camera User takes a photowith the digital camera and then selects “Send to Netpage” from a menu.This results in the photo being added to the Netpage clipboard (say withtype: image/jpeg). mp3 audio players User chooses a favourite song (orsong collection) on the audio player and selects “Send to Netpage” froma menu. Video camera Similar to the digital camera scenario, but forvideo instead of still photo. Dictation device User records a messageand then pushes the audio file to the Netpage clipboard.4.2.3 Implicit Push

To improve usability, it may be possible for certain devices to supportimplicit (or auto) push which is where certain user interactions with adevice cause a value to be automatically pushed to the Netpage clipboard615. As an example, the user may configure their digital camera so thattaking a photo causes the photo to be pushed to the Netpage clipboard615.

4.3 Netpage Clipboard Interface

In order for a device or application to push a value to the Netpageclipboard 615 or retrieve the current object 601, the device orapplication first retrieves a reference to a NetpageClipboard object forthat user from the Netpage server 629. The following listing shows theNetpageClipboard interface in its most basic form. The interface allowsthe current clipboard value to be set, fetched and cleared.

interface NetpageClipboard   {     void     setObject(in PlayRequesta_object);     PlayRequest     fetchObject( );     void     clear( );  };

The sections that follow discuss alternatives for various aspects of theNetpage clipboard interface.

4.3.1 Multiple Values Per User

The Netpage clipboard 615 is able to support multiple current valueseach of which belong to a different category. For example, the clipboard615 can hold a current audio and current video at the same time.PlayRequest values can specify one or more categories to which thatvalue belongs. Note that an object category is generally independent ofthe specific physical type of the value being added to the clipboard.For example, setObject( ) might be called with a value that has acategory of “photo”, and a physical type of image/jpeg.

This listing shows a NetpageClipboard interface that supports multiplecurrent values.

interface NetpageClipboard   {     void     setObject(in PlayRequest);    PlayRequest     fetchObject( );     PlayRequest     fetchObject(inObjectCategory);     void     clear(in ObjectCategory);     void    clearAll( );   };

Such a clipboard would still support the current object concept whichwould return the most recently added object. To that end, the clipboardinterface in the listing has two fetchObject( ) methods. The first takesno parameters and returns the most recently added object. The secondtakes an ObjectCategory as a parameter and returns the most recentlyadded object which belongs to the specified object category.

More generally the clipboard could reuse the general capability matchingmechanism required by RequestRouters in the Netpage Player architecture.This would provide the clipboard with a very powerful interface forretrieving PlayRequests.

4.3.2 Multiple Representations of Values

As with clipboards in standard desktop environments, it may be desirableto allow the Netpage clipboard to support multiple representations ofvalues, each of which would have a different MIME type. For example, atext object could be stored as both a text/plain and a text/rtfdocument.

4.3.3 Using Values from the Netpage Clipboard

Once a user places a value in the clipboard, the value can be accessedby applications in response to actions by the user. The basic clipboardinteraction model is shown in FIG. 47. Referring to FIGS. 47 and 48, theclipboard starts empty 652. A value can then be pushed 653 into theclipboard 615. At that point an operation can be selected 654 by theuser at which point the selected operation is executed against thecurrent value in the clipboard. If an operation is selected while theclipboard is empty, then an error 655 is returned to the user.

The model presented in FIG. 47 is called the value first model as itrequires the user to first select the parameter to an operation, and tothen select the operation itself.

One way in which an operation can be invoked by a user is for the userto interact with a printed Netpage form 660 which contains a set ofPlayRequests which specify operations. FIG. 48 shows such a printedcommand sheet and Table 6 describes the meaning of each operation. Whena field is selected by the user (by clicking on it with a Netpagepointer), the corresponding operation is performed on the valuecurrently stored in the clipboard

TABLE 6 Description of commands Command Descriptions Play Executes thedefault action for the current clipboard value. For example, for a phonenumber object, the default action might be to dial the number, while foran image it might be to display the image on a device capable of imagedisplay. Display Display the current clipboard value. This is differentto play, since for a phone number, for example, the phone number isdisplayed rather than dialled. Attach to Page Associate the currentclipboard value with a location on a printed page. Print Print thecurrent value.4.4 Placing Operations and Targets in the Clipboard

As clipboard entries are actually stored as PlayRequests 521 it is alsopossible to push operations and targets to the clipboard 615.

4.4.1 Adding Operations to the Clipboard

FIG. 49 shows a clipboard model which allows operations 523 to be storedin the clipboard 615 rather than values. In this model, the user pushesan operation 523 to the clipboard 615 at step 661, and then selects thevalue 524 which the command 523 should be applied at step 662. So, forexample, a “play” operation could be pushed to the clipboard 615, andthen a phone number could be subsequently selected from a contacts listas already shown in FIG. 46. This model is called the operation firstmodel as it requires the user to first select an operation 523, and tothen select the parameter 524 for the operation 523.

4.4.2 Allowing both Value First and Operation First Models

It is possible to simultaneously support both the value first andoperation first models as shown in FIG. 50. In this model an emptyclipboard 615 allows an operation 523 or a value 524 to be pushed atsteps 663 or 664. This gives the user the freedom to perform operations523 in whichever order seems natural. Also, if a user is using operationfirst, then once the user has placed an operation 523 in the clipboard615, they can perform that operation 523 on a many values 623 asrequired simply by pushing values to the clipboard 615 at step 665(andvice-versa for value first at step 666). In order for a user to switchbetween models, however, the clipboard 615 must be explicitly cleared atsteps 667 and 668.

The following examples show the two models in action. In each case, theuser's steps are shown numbered and the contents of the clipboard 615are shown after each step. In the first case, the user uses the valuefirst model by first pushing a phone number to the clipboard 615, andthen selecting the operation 523 to be applied to that object:

-   -   Push value    -   value {phone-number, “555 1287”}    -   Push operation    -   value {phone-number, “555 1287”}    -   operation {play}

At this point, an observer of the clipboard 615 (likely theUserRequestHandler) determines that the PlayRequest 521 as shown belowcan be produced by combining the operation 523 and value 524PlayRequests 521. The resultant PlayRequest 521 can then be routed bythe UserRequestHandler. Typically the UserRequestHandler “plays” aphone-number by sending the request to a device capable of dialling thephone number.

-   -   PlayRequest created by merging contents of clipboard:

target operation play values phone-number “555 1287”

The second case uses the operation first model:

-   -   Push operation    -   operation {play}    -   Push value    -   operation {play}    -   value {phone-number, “555 1287”}

At this point, the clipboard 615 determines that the PlayRequest 521 canbe produced by combining the operation 523 and value 524 PlayRequests521.

4.4.3 Ambiguous Usage

One problem associated with this mechanism of pushing values 524 oroperations 523 to the clipboard 615 is a value 529 or operation 523 canremain in the clipboard 615 indefinitely. As such, it is easy for a user525 to forget that an object 601 is in the clipboard 615 and forunexpected results to emerge. This is particularly the case if wesimultaneously allow both the value first and operation first models. Asan example, suppose the user performs the following actions:

-   -   1. Push “play” operation to the clipboard    -   2. Push a phone number (the number is dialed)    -   3. An hour later, click on another phone number.

Using the state machine indicated by FIG. 50, step (3) would cause thenumber to be dialed. This may not be what the user expected as they havelikely forgotten the fact that the “play” operation is currentlyresiding in the clipboard.

There are a number of approaches to addressing these useabilityconcerns:

-   -   Single Use Clipboard Entries    -   Clipboard Timeouts        4.4.4. Single Use Clipboard Entries

FIG. 51 provides an alternate model to that previously described. Inthis model, using an item in the clipboard 615 at either of steps 525 or626 results in that item being removed from the clipboard 615 at state670. That is, operations 523 and values 524 only remain in the clipboard615 for a single use after which the clipboard 615 is returned to theClipboard Empty state 627. This model largely avoids the useabilityconcerns described above, although not completely, as will be describedin further detail.

4.4.5 Clipboard Timeouts

While a single-use clipboard model improves the useability of theNetpage clipboard 615, there are still problematic scenarios whichresult from allowing both value first and operation first models tocoexist. Consider the following steps:

-   -   Push “play” operation to the clipboard 615    -   An hour later, push a phone number

In the above, it is not clear whether the user 525 has indicated thatthey would like to apply the “play” command to the phone number, orwhether they had forgotten that they had pushed the “play” operation anhour ago and were actually attempting to simply push a phone number tothe clipboard 615.

In order to address the ambiguity, the concept of clipboard timeouts canbe introduced, as shown in FIG. 52. In this model, objects only remainin the clipboard for a limited duration, 60 seconds in the example forsteps 675 and 676, but the exact value could be user configurable.

FIG. 52 shows clipboard timeouts in the context of a single-useclipboard. It is also possible to introduce clipboard timeouts within amulti-use clipboard, as shown in FIG. 53. Each application of anoperation to a value at steps 680 and 681 resets the timeout periodwhich allows for multiple values to be applied to an operation withouthaving to push the operation each time. Similarly, it allows multipleoperations to be applied to a value without having to push the valueeach time.

4.4.6 Multi Value Operations

The clipboard 615 concept can be extended to support operations thatrequire more than one parameter 524. The basic approach is to allowmultiple values 524 to be pushed to the clipboard 615. The clipboard 615can then combine the values 524 with a pushed operation 523 to create aPlayRequest 521 with multiple parameters.

4.4.7 Adding Targets to the Clipboard

Consider the command sheet already shown in FIG. 48. Clicking on eachoperation with a Netpage pointer 533 causes the corresponding operation523 to be pushed to the clipboard 615. The operation 523 does notnecessarily take effect immediately. The operation takes effect whenthere is sufficient information available in order to determine the fulldetails of the PlayRequest 621 being requested by the user 525. Thereare cases in which it proves valuable to allow the user 525 to be ableto specify the target 522 of an operation 623. For example, suppose auser wishes to print a photo from their digital camera, but does notwish to print it to their default printer.

Selection of the printer can be achieved by selecting the printer from alist of printers printed on a Netpage card 700 as shown in FIG. 54.Clicking on a printer 707 on the card simply causes the details of thatprinter to be pushed to the Netpage clipboard 615.

FIG. 55 shows the details of each of the fields on the card. Each fieldcorresponds to a PlayRequest (521 a, 521 b, 521 c, 521 d, 521 e, 521 f,521 g) that specifies a target 522 and nothing else.

The following steps provide an example in which the user pushes a targetto the clipboard.

   1. Push photo of family dog to clipboard value { image/png category=”photo”, <contents of photo of dog> }    2. Select printer by clickingon card value { image/png category =”photo”, <contents of photo of dog>} target { home-color-printer }    3. Push “print” command to clipboard(e.g. using printed command    sheet shown in FIG. 3 value { image/pngcategory =”photo”, <contents of photo of dog> } target {home-color-printer } operation { print }

At this point, a PlayRequest 521 can be constructed that combines allthe elements from the clipboard 515 as shown in FIG. 52.

PlayRequest created by merging contents of clipboard targethome-color-printer operation print values image/png category =”photo”<contents of photo of dog>

Once the full PlayRequest 521 is determined, it is performed and,depending on the clipboard model being used, the clipboard 615 wouldeither be cleared of all contents (the single-use model), or left as isin readiness for future related requests 521 (the multi-use model). Inthe latter case, the subsequent act of pushing another photo (say of thefamily cat) to the clipboard 615 would leave the clipboard 615 in thefollowing state:

target { home-color-printer } operation { print } value { image/pngcategory =”photo”, <contents of photo of cat> }

At this point, the clipboard contents indicate that the user wishes toprint the newly selected photo on the color inkjet printer at home. Thisapproach allows the user to request the printout of a number of photoson a particular printer, without having to specify the target 522 or theoperation 523 each time, by simply clicking on each required photo.

Even in the multi-use case, if the user is inactive for someconfigurable period of time, then a clipboard timeout causes theclipboard 615 to be cleared.

4.4.8 Pushing more fully Specified PlayRequests

As the clipboard 615 supports the pushing of PlayRequests 521, theapplication author is not limited to pushing values 524, operations 523,and targets 522. It is also possible to push PlayRequests 521 that aremore fully specified. For example, consider the printed command sheet710 shown in FIG. 56. It contains various commands 711 that can beinvoked by clicking on the sheet with a Netpage pointer 533.

FIG. 57 shows the configuration of one of the commands. Namely, the“print using . . . office printer” command 712. As can be seen, itcorresponds to a PlayRequest 521 that specifies both a target 522 and acommand 712.

  1. Push photo of family dog to clipboard value { image/png category=”photo”, <contents of photo of dog> }   2. Click on “print using ...office printer” field value { image/png category =”photo”, <contents ofphoto of dog> } target { office-personal-printer } operation { print }

At this point, a PlayRequest 521 can be constructed that combines allthe elements from the clipboard 615 as shown in FIG. 52. As such, the“print using . . . office printer” command 711 has reduced the number ofclicks required by the user to perform the action from three clicks totwo.

PlayRequest created by merging contents of clipboard targetoffice-personal-printer operation print values image/png category=”photo” <contents of photo of dog>4.4.9 Communicating Status to the User

At times the mechanisms described may require communication of statusinformation (often errors) to the user. One method is to make use of theNetpage Player infrastructure. Special operations (eg.show-status-ok-message and show-status-error-message) can be designatedfor transmitting status information to the user. The player architecturewould determine, for each message, the appropriate device (or devices)on which to display the message and the way in which to display it. Forexample, it may be that in certain situations the pointer 533 is theonly available player in which case an error status might be “played” byilluminating a red LED on the pointer 533 or playing a short sound.

5. Downloadable Content Billing

5.1 Overview

There is already an existing market place for purchasing and downloadingproducts to mobile devices. Products such as: ringtones (monophone,polyphonic and real tone); wallpapers; games and other applications;music; music videos; films and TV. The printing capabilities of M-Printcan add further to this list of products.

The traditional methods for purchasing and downloading such products arebased around web browsing and SMS to initiate the download and EMS orMMS to deliver the product to the phone 100, see FIG. 58. The userenters and sends a product code via SMS at steps 720 and 721 to a vendor723 who then delivers the product to the originating handset 100, or tothe mobile number supplied with the product code at step 724.

Netpage and M-Print technologies have the ability to simplify the userexperience in accessing and purchasing these products, while also beingable to utilise the existing infrastructure for the billing and deliveryof the products.

FIG. 59 shows the typical sequence of events for a Netpage playsequence, cast in terms of downloading a preview of a ringtone. In thegeneral Netpage play sequence a Netpage click at step 730 triggers aform submission which in turn results in a general “play” event beingrouted at steps 732 and 734 to a Netpage player via the Netpage PlayerArchitecture. For downloadable content the form submission is shown as asimple hyperlink 731 and the “play” event is shown as a download ofcontent, in this case a ringtone.

Using a Netpage pen or pointer 533 the user 525 can click on a printedadvertisement 715 in a magazine, newspaper, direct mail, on a product'spackaging or possibly on a product itself. This can automaticallydeliver some content to the user's preferred Netpage Player 520 for thatcontent. Most likely the Netpage Player 520 is the user's mobile phone100, as is shown in FIG. 59. Alternatively it can commence a dialoguewith the user via the device's UI to determine what the user wants to dowith the product they selected via their click. For content likeringtones or phone themes, this provides a more convenient interfacethan the existing SMS interfaces in use today.

M-Print devices can provide a Netpage scan operation, and this can alsobe used to initiate a product download or purchase as above. The cardthat is scanned could have been printed on a M-Print printer 4 or itcould be provided along with another product, e.g. a card in a breakfastcereal box.

Combining the abilities of M-Print and Netpage can lead to a powerfulsuite of new product promotion and sales tools. For example, a user canuse a Netpage pointer 533 in a mobile phone 100 to click on the latestringtone advertisement in their favourite magazine. This results inpurchasing the ringtone, charging it to their mobile phone account,download and install it on their phone 100. It can also print on theirphone a promotional card that allows a one time download of the sameringtone for them to give a friend. When their friend scans the cardthey receive the same ringtone downloaded and installed on their phone.Depending on the promotion running at the time the friend may receivethe ringtone for free or at a discounted rate.

5.2 Downloading Content

FIG. 59 shows the typical Netpage player sequence of events that areinvolved in downloading a product, in this case a ringtone, from aNetpage based application. In the diagram the ringtone download eventbeing routed to the phone in steps 732 and 734 consist of a ringtone,with the play operation of preview 716. It may have the target handsetspecified or it may rely on the Netpage player request routingmechanisms to determine the appropriate target 522 for the play request521.

It has already been noted that the mobile computing environment alreadyhas some well established mechanisms for satisfying on-line productpurchases. Below are three possible ways Netpage player requests can beintegrated into the existing mechanisms to provide the user with asimpler user experience. In each of the cases the request to purchase orpreview the product is instigated via a Netpage stroke, click or scanThe differences arise in how the Netpage application 733 procures theproduct and delivers it to the user.

5.3 Netpage as Middle Man

FIG. 60 shows a scenario where the Netpage ringtone application is usedas an alternative interface and delivery mechanism to an existingringtone vendor 723. The Netpage application 733 does not own the rightsto the products it is providing. Instead it is acting as a middleman,forwarding the requests at step 736 onto the product vendor 723 onbehalf of the Netpage user and routing the requested product at step 739back to the Netpage user via the Netpage player mechanism.

This approach allows the user to benefit from the Netpage PlayerArchitecture, which allows the user to customize the way they want theproduct to be routed after they have received it from the vendor. Italso allows for the Netpage Player to provide extended handling ofproducts on the mobile device. For example, the player may prompt theuser if they wish to back up their existing ringtone before installing anew ringtone or it may guide the user through installing the ringtone asa custom ringtone for a particular phone number or set of phone numbers.

5.4 Netpage as a Sales Agent

FIG. 61 shows the Netpage application 733 acting as a sales agent forthe product vendor 723. The play request 521 contains the userinformation and the product ID, and is routed to a special player 743 atstep 741 that passes the request onto the vendor 725 at step 742. Thevendor 723 then delivers the product in its normal way, in this case viaEMS/MMS at step 744.

5.6 Hybrid Approach

FIG. 62 shows a hybrid scenario where the both the Netpage PlayerArchitecture and the existing mobile technologies are used to deliverthe purchased product. The request 521 to purchase the product may besatisfied by the ringtone application 733 either internally as a pureNetpage application or acting as a middle man. The play request isdispatched containing both the purchased product and a voucher to beprinted.

The play request 521 is routed as a Netpage player request and when itreaches a player 745 deployed within the mobile network, acting a mobiledevice proxy, it is split into a product delivery and a print job. Theproduct is sent out via an EMS/MMS message at step 746 and delivered tothe mobile device 100 in the same way as a traditionally purchasedproduct at step 747. The print job in the form of an M-Doc reference 507is delivered to the phone 100 via one of the mechanisms initiating aprint job on the mobile device 100.

5.7 Billing for Downloaded Content

The ability to bill or charge for services and products is at the heartof all businesses. To be able to bill you need to be able to:

-   -   1. identify a party to bill;    -   2. have a mechanism to deliver and ensure payment of a bill;        and,    -   3. be able to justify the bill based on records of services or        products provided.

When accessing a service or purchasing a product over a mobile networkthe user is typically charged for both the data or call traffic involvedin making the transaction and also for the product or service purchased.For example, if a ringtone was purchased via Netpage, there is a chargefor the ringtone and also a charge the data traffic used to download thecontent to the mobile device. It is possible for a 3rd party vendor toenter into a commercial relationship with the carrier whereby thecarrier waives or reduces the data transfer costs, in return for apayment from the vendor. Mobile carriers play a central role in billing.They own the private networks used by mobile devices to connect to boththe public telephone and data networks and it is at their discretionthat a mobile device has access to any services. They already have abilling relationship with each of their customers and hence are able toidentify them and bill them. Over time the billing options offered bycarriers have evolved from the simple post-pay phone bill to includepre-paid, post-paid and plan-based billing, where customers arecommitted to paying a certain amount per period for which they canaccess a range of services. Carriers also have the ability to adjust orwaive billing based on a user's activity over a billing period, forexample, if you send more than certain number of SMSs in a day then allSMSs is to be charged at a different rate.

For pre-paid accounts the account balance is checked before thetransaction is commenced to ensure it has sufficient credit to pay forthe service. Not all services can predict the total cost before they aredelivered and in those cases it is up to the individual carriers todecide whether those services are to be made available to pre-paidcustomers or not.

Mobile carriers recognise the value of their ability to identify andbill customers and make it available to third parties on a commercialbasis. They do not make it freely available. The ability to integrateinto a carrier's billing system is typically offered in a number ofdifferent ways, of which some are:

-   -   1. SMS/MMS-based services    -   2. Hosting of 3rd party applications and billing the traffic for        those applications differently    -   3. Billing of data traffic to nominated servers at different        rates    -   4. The ability to bill on behalf of a third party, based on        billing records provided by the third party.

To bill for products or services attained via Netpage interactions it isnecessary to identify where in the sequence of events it makes sense togenerate billing records and which party in the transaction isresponsible for maintaining and acting on those billing records. To makethese services available to users with pre-paid mobile accounts it maybe necessary to be able to predict ahead of time the total cost of thetransaction.

If we consider the SMS-based content download scenario, FIG. 58, thereare three billing opportunities:

-   -   1. The SMS to initiate the transaction may be a billable SMS    -   2. The purchase of the content from the vendor    -   3. The delivery of the content via a SMS or MMS.

The first SMS is a standard SMS that the user may be billed for, it maybe billed differently given that it is destined for a vendor who has abusiness relationship with the carrier. The second SMS/MMS is not billedto the sender, as would normally happen, but it is billed to thereceiver, via an agreement between the vendor and mobile carrier. Inboth of these cases the mobile carrier generates and manages the billingrecords. The purchase cost for the content is also billed to thereceiver and the vendor is responsible for generating the billingrecord. In most cases this billing record is forwarded onto the mobilecarrier, typically once a day via a batch transfer. The carrieraccumulates these records and includes them on the user's phone bill asa service for the vendor. The carrier then makes a periodic payment offunds collected on the vendor's behalf to the vendor. This alleviatesthe need for the vendor to establish a billing relationship with each ofits customers and also allows the mobile carrier to derive more revenuefrom its existing billing relationship with the customer.

For the vendor to be able to generate billing records that a mobilecarrier can incorporate into a customer's bill the mobile carrier needsto provide the vendor with a “customer id” or “billing id” pertransaction that the carrier can use to link a billing record with acustomer. The provision of customer id and the ability to generatebilling records needs to be done in a secure way to reduce thepossibility of fraud. To ensure the required security is maintained eachvendor enters into a contractual relationship with each mobile carrierbefore the “customer id” data is shared with them.

In situations where a vendor cannot or does not want to work with amobile carrier they need to implement their own means of identifying andbilling their customers. Identifying a user of a mobile device withoutthe assistance of the mobile carrier in a uniform way across all mobiledevices is non-trivial. Introducing Netpage and M-Print technologiesprovide a number of new billing opportunities for all parties involved.If we consider the case where Netpage acts a middle man, there are twobilling opportunities:

-   -   1. When Netpage application retrieves the content from the        content vendor;    -   2. For the network traffic used to deliver the click and the        ringtone.

The Netpage application fetching the content from the vendor generates abilling record for the customer. As for the SMS case above, it ispossible to arrange with a mobile vendor to accept these billing recordsand to bill on the application's behalf. To do this the data trafficassociated with the hyperlink activation is to be tagged with a“customer id” to be associated with the billing record.

The mobile carrier tracks the data traffic used to send the click eventand deliver the play request. By default that traffic would be billed inthe same way as all other data traffic to and from the device. In asituation were the Netpage application is billing via the carrier it maybe possible to strike a deal with the mobile carrier where by they waivethe data traffic costs in return for a payment per transaction.

In the case of Netpage acting as a Sales Agent, the Netpage system isnot involved in delivering the product but only in making the sale. Inthis case there are four billing opportunities:

-   -   1. The data traffic for the Netpage click;    -   2. The forwarding of the purchase request to the vendor;    -   3. The purchase of the content;    -   4. The delivery of the content

The mobile carrier tracks and generate billing records for the datatraffic and the delivery of the content. As in the SMS case the deliverymessage is charged to the receiver rather than the sender, via anagreement with the carrier.

In this case the billing record for the content purchase generated bythe ringtone vendor and would be handled in same way as for the SMS caseabove. It would be possible to have an agreement with the carrierwhereby the cost of the data traffic and the delivery message are waivedin return for a payment from the vendor.

In most cases the Netpage application would be managed by the vendor asan alternative interface to their existing business and as such it doesnot need to bill for its forwarding services. If it is run by adifferent business then it generates a billing record for the forwardingservice. This may be delivered either to the user or to the vendor whohas agreed to pay for forwarded requests.

The billing options for the hybrid approach are essentially the same asfor when Netpage is a middle man but the number of deliveries, and hencebillable events, to the mobile device is increased. In this case thebill for delivering the M-Doc may be changed to the vendor rather thanthe customer since it is an unsolicited promotion. Each of thesebillable events could be filtered by the mobile carrier based on laterbilling records that enter the system. For example, the cost ofdelivering the M-Doc may be waived or credited back, if the offer on theprintout is taken up by another user.

5.8 Digital Rights Management

Closely related to billing is ensuring that the product or service isonly used for the purpose(s) it was purchased for, eg. a ringtonepurchased for one phone, can not be installed on more than one phone, ora music file that is downloaded as a sample can only be played X timesbefore being purchased.

Digital Rights Management (DRM) schemes are being adopted by mobiledevice manufactures and mobile carriers. Where content is beingdelivered via the existing mobile mechanisms DRM is automaticallytriggered. The Netpage Player on a mobile device is implemented to hookinto the DRM mechanisms on the mobile device. It prevents a userby-passing DRM restrictions on downloaded content.

5.9 Identifying the User

Mobile carriers identify the user of a mobile device when the devicenegotiates access to the network, either at boot time or when it comeswithin range of a base station. For GSM and CDMA networks a user'sidentity is determined by matching the International Mobile SubscriberIdentity (IMSI) with a user's records held by the carrier. The IMSI isnot sent during negotiation but rather an identity derived from itcalled a Temporary Mobile Subscriber Identifier (TMSI) is sent. Accessto the IMSI can be protected by a PIN and most phones can be set up toprompt the user for a PIN when it is turned on. This is used to gainaccess to protected data, such as the IMSI. During the connection timenegotiations the International Mobile Equipment Identity (IMEI) is alsotransmitted. This is not used to determine who the user is, but it isused to filter out mobile devices that are blocked from the network,e.g. stolen mobiles can be blocked based on their IMEI.

When a mobile network allows a user on a GSM network, they assign theuser a SIM card and then activate the SIM card. For a non-pre-paid SIMcard the user needs to present sufficient identification informationthat the carrier is satisfied they know who the person is, that they areable to pay their bill, and where to send the bill. For a pre-paidaccount the carrier does not need such information, but some countriesdo require carriers to collect user identity information even forpre-paid accounts. Once the user has an activated SIM card it can beused from any handset and the correct account is billed for usage. Usersare encouraged to protect their SIM cards with a PIN, so that it cannotbe used until a PIN is supplied. However this is not enforced by thecarriers, as it is a user choice.

The Netpage system, includes the concept of a Netpage user, each of whomhas a Netpage account. Some Netpage applications require knowledge ofthe user. To be able to access those applications via a Netpage pointeror scanner built into a mobile device, a link between the mobile deviceand a Netpage user is established. To do this there is a configurationand activation step, similar to a SIM card activation where the Netpagesub-systems on the mobile device are configured and a mapping is setupbetween the user identity for the mobile device and a Netpage user. Ifthe Netpage account is being established for the first time then, aswith a mobile carrier, the user identifies themselves so that theNetpage network operator is able to bill the user for any Netpagerelated costs incurred.

There are times where a mobile user's identity cannot be determined viaa mobile carrier: possibly a company does not want to or cannot get theinformation from a carrier; the mobile device may be connecting to thenetwork without a carrier being involved; or the carrier may not havethe information required, eg. anonymous pre-paid accounts. In thesesituations the functionality available to the user can be reduced tofunctionality that does not require knowledge of the user's identity oradditional Netpage specific information can be used to establish theuser's identity to the satisfaction of the Netpage system.

Netpage-specific user identity information is stored on the deviceduring the Netpage registration/activation process. To give the samelevel of user identity portability as a SIM card the information can bestored on the SIM card, if it is present. If that is not possible, dueto a mobile carrier denying access to it or it not being present, it canbe stored in a secure store within the device and if that is notpossible, on the normal file system of the device. If the identityinformation is stored on the SIM card, it can automatically move withthe user's carrier identity when they swap SIM cards.

6. Use Cases

6.1 M-Print Blanks

The M-Print printer 4 is able to print on special M-Print blanks thatare specially designed to provide optimal print quality in a M-Printprinter 4. The blanks are purchased by a user in packs and loadedone-by-one into the printer when a print is being made.

To ensure valid blanks are loaded into the printer 4, a mechanism forvalidating the supplied blanks and rejecting imitations can be supplied.This can be done by reading an ID from the blank during printing andvalidating that ID either locally or via a network service.

The blanks are pre-tagged. For Netpage to be able to correctly registera printout an ImpressionID is determined from the pre-tagged blank whenthe printout is printed. This implies the M-Print printer is able toread the ImpressionID from the blank.

Both cases can use the ID encoded on the blank. The proposed scheme forvalidating the ID involves reading a second number from the blank calledthe signature. The signature and ID can then be validated as a pair. Theproposed mechanism for this is to consult a network based service thatsecurely stores the ID and signature pairs that have been manufactured.

Both the ID and the signature are readable by the printer 4 and in theNetpage case by a Netpage pointer 533. The M-Print printer 4 does notcontain a Netpage pointer 533, but it includes a bar code reader. Thismeans the ID and the signature are provided on the back of the blank asa linear bar code, most likely in IR ink, for the printer 4 and on thefront of the blank encoded in the Netpage tag encoding for the Netpagepointer 533.

In some circumstances, validation of the ID may not be possible in realtime before the printout completes. Thus, users are informed ifnon-genuine blanks are being loaded into the printer and warn the userthat loaded non-genuine blanks may decrease the lifetime of theprinter's printhead.

While complete validation of the ID may not be possible before printing,coding on the blank can be detected that indicates the orientation ofthe blank and also the start of the timing codes that allow the printer4 to detect the speed at which the blank is moving through the printer4. If these can not be detected the blank may be rejected beforeprinting commences.

6.1.1 Loading a Card

Referring to FIG. 63, a print job has been submitted to the printer 4,before it can commence it must be supplied with a valid blank the rightway up.

-   -   1. The user is prompted to insert a blank (step 750)    -   2. The user inserts a blank (step 751)

If the user inserts the paper upside down at option 752, the user isprompted to re-insert the blank the other way. If the loaded paper isuncoded or incorrectly coded, the user is prompted to insert a genuineM-Print blank.

6.1.2 Validate an ID

Referring to FIG. 64, during or immediately after printing the printer 4sends back the ID and possibly the signature of the blank at step 750.In the background the M-Print services on the mobile device validatesthe ID at step 757, either locally at step 759 or via a network serviceat step 758, and if the validation fails it informs the user at step763. There is a timing issue here in that the user may no longer belooking at the mobile device 100 when the result of validation is known,to alert the user that the blank they have printed on is not valid andthat using that media shortens the life of the printhead the M-Printservice on the device can deliver the message as a local SMS causing thedevice to alert the user of a new message.

Once the ID and signature have been validated it is passed at step 762onto a Netpage microserver 761 for processing as the ImpressionID of theprintout. The user is alerted that the last print was done on an invalidblank and continuing to use such blanks shortens the life of the printer4.

6.2 Printing

Common to all the use cases present below is printing. From a usersperspective, printing normally has two stages, a third may be added ifan error occurs or the blank is invalid:

-   -   1. Loading the blank    -   2. Printing    -   3. Error reporting

The user view of loading the blank and reporting an invalid blank arecovered above. The user's view of printing is both: a progress dialogthat allows the user to view the progress of the print and cancel theprint; and being able to see the blank move through the printer andemerge from the device.

Canceling the print stops the printer using any more ink, but the ID andsignature from the blank may still be read for validation.

6.2.1 Print

-   -   1. The user is prompted to load a card.    -   2. A print progress dialogue is displayed, with a cancel button    -   3. The blank is drawn fully through the printer and the print        progress dialogue is removed

If the user wants to cancel the print job the user presses cancel on theprint progress bar which results in the print job stopping and the blankbeing ejected. If an error occurs such as a paper jam, an error messagemay be provided to alert the user. The user may then dismiss the dialogof the error message and progress dialogs are removed. If a blank failsvalidation, the user is alerted of the failure.

FIG. 65 shows a Printer DC 770 as the object that exposes the graphicsmodel used by applications 514 to build up the printed image. It showsthe printing following after the application 514 has completed drawingthe page. On some systems these operations may overlap or theapplication 514 may be requested to draw the page multiple times withdifferent clipping regions, eg. rendering in bands, either way thelogical flow should still be the same. If capturing all printeddocuments as Netpage documents is supported then fully composed page islodged with the Microserver 761, the Microserver 761 treats this adocument lodgement, and records a printout for that document, when theID and signature are successfully validated and passed onto themicroserver 761.

FIG. 65 shows the ID and signature being passed out from the printer 4soon after printing has commenced, the timing of the transmission of theID and signature is not significant for the user since they only knowabout it if the validation fails which occurs after the print hascompleted. For Netpage enabled printouts, this timing is more critical,since a situation can occur where a printout is immediately handed toanother user who clicks on it with a Netpage pointer or pen 733 andexpects a result. Where the printing device does not have networkconnectivity this may not work at all. If the printing device does havenetwork connectivity it still might not work, or at least have aperceivable latency, while the network validation and the ID andregistration of the printout is completed.

6.3 Uploading and Downloading Data

Moving data on and off mobile devices via the wireless network reliablypresents a number of challenges:

-   -   1. Wireless networks are inherently unreliable and the link can        be lost at any point. This can occur due to signal loss or the        mobile device having to disable the radio link to conserver        power or to allow another power hungry activity to start up, eg.        printing.    -   2. The bandwidth available on the existing 2.5G networks is        limited and it could take up to several minutes to transmit a        high resolution photo on or off the device    -   3. The cost of transferring data over a mobile network can vary        and it may be significantly cheaper to delay the transfer to a        non-peak time, e.g. late at night.    -   4. Mobile carriers might prefer non-urgent data transfers to        happen during lulls in the network traffic.    -   5. It may be cost effective to support different means of        transferring data. A HTTP Put over GPRS is the most obvious way,        but it may be possible to take advantage of carrier price        subsidies and send the same data via an MMS or SMS message.

Referring to FIGS. 66 and 67, to provide flexibility in how datauploading and downloading is achieved and to insulate the rest of thearchitecture from it, each mobile device 100 has a service responsiblefor delivering and receiving data from the network. It queues datatransfer requests 780 and guarantees delivery of the data in the anefficient manner. It supports partial transfers and resumption oftransfers after a break in the data link to minimise the data sent overa wireless network. An equivalent service can be located in the networkto both receive data from the mobile device and forward it on to itsdestinations and also to queue data being sent to the mobile device.

The network based component of this service provides a carrierintegration point. A carrier may choose to host this service and billthe traffic for it differently to encourage usage of the M-Print andNetpage services. It also provides a single point of modification totake advantage of new features in the carrier networks, eg. a new datapush model not based on SMS.

The device based component of this service provides a simple interfaceto the device based M-Print and Netpage services, while providing theability to exploit all the features of a mobile device 100 to accessnetwork based services. This may include taking advantage of WLANconnectivity where possible.

The download sequence shown in FIG. 67 illustrates how “data push” to amobile device can be achieved which cannot be directly reached from thegeneral internet. It shows a SMS message being sent to the device toinform the download service that there are downloads waiting to befetched. The message may include some information about the urgency ofthe downloads, to allow the device side download service to decide whenit should fetch the download(s).

6.4 Netpage Pointer/Scanner

In the printing scenario a blank's ID is read during printing. This IDis used to both facilitate validation of the blank as a valid M-Printblank and also as an Impression ID for Netpage. The Impression ID isused by the Netpage server 529 to associate the M-Print printout withthe document that was printed onto the blank.

An M-Print blank has the ID encoded on the back of the blank in a waythat is readable by the paper feed mechanism. If the blank has beenpre-tagged with Netpage tags then the ID is also embedded in the Netpagetags. To initiate a Netpage interaction the first step is for the userto perform an action that retrieves the ID and supplies it to a Netpageserver. The ID can be retrieved by:

-   -   1. A stroke or click with a Netpage Pen;    -   2. A click with a Netpage Pointer;    -   3. Scanning the ID on the back of the card.

The first two of these actions use Netpage specific devices to read theNetpage tags. The pen 533 provides a stream of digital ink along withthe ID and the pointer provides a position on the printout along withthe ID. The last uses a scanning mechanism to read the ID from the backof the printout. It can be the same scanner as is used in the M-Printprinter paper feed mechanism or it can be a dedicated scanner that theprintout is feed into or passed over, similar to a bar code reader. Thismechanism only provides the ID, it does not provide any positionalinformation, but Netpage applications can be authored to support a“scan” of the printout as well as a click or a stroke on the printout.

6.4.1 Activate a Netpage Application via a Scan

The user passes an existing M-Print printout back through their M-Printprinter to activate the associated Netpage application.

The printout is drawn through the printer and a Netpage application islaunched. If there is no Netpage application for the printout, the useris told the printout has no associated application. If the printoutfails validation, the user is alerted of the failure.

The effect the user sees as a result of a Netpage application beinglaunched varies depending on the application, some examples are:

-   -   For a photo the use may see the photo displayed on the screen        with the option to reprint it or send to someone.    -   For a business card the user may receive a vCard on their mobile        device which can be processed in the normal way.    -   For a coupon a completed SMS/MMS/email may be displayed asking        the user if they wish to send it off to enter the competition.    -   Any printout may be displayed on the mobile device showing the        print image and allowing the user to navigate the hyperlinks and        fields in the printout and activate them. If the device has a        touch screen, the user can use a pointer to select fields and        generate digital ink.

FIG. 68 is a sequence fragment that shows the processing of a scanned IDfrom the paper feed mechanism in the printer. If a scanned ID andsignature is returned to the Print Service while it is printing, seeFIG. 65, it is validated and then passed onto the Netpage Microserver761 triggering the submission of the Netpage document. The validation isperformed by the Print Service 754 in this case to ensure the user canbe warned about using invalid media as early as possible. If the PrinterService receives a scanned ID and signature while it is not printing atstep 790 then it passes it directly to the Netpage Microserver 761 as apseudo-click or scan. The Microserver 761 processes it in a similar wayto the way it processes a Netpage pointer click.

7. Applications

7.1 Photo Printing

Photo printing is a major application for M-Print. In its simplest form,photo printing can be done completely locally, without any dependence onnetwork services or interactions. Printing a photo can interact with aphoto archive. If a photo is printed then it is likely the user may wishto access or print the photo again. When a photo is printed it can bepushed out to the photo archive making it available for on-lineretrieval or access. Netpage functionality offers a convenient andnatural way of interacting with a printed photo. A Netpage enabled photocan act as a permission token, giving the holder of the printoutpermission to retrieve and reprint the photo from the archive. Photoarchiving from mobile devices is an independent application from photoprinting. Users typically take more photos that they wish to keep in anarchive than they wish to print.

7.1.1 Local Photo Printing

Referring to FIG. 69, a camera phone user takes a photo at step 795 andelects to print it.

-   -   1. The user uses the default photo application 796 on their        device and takes a photo at step 795    -   2. The user selects print from the applications menu    -   3. The user is prompted to load a blank    -   4. The print is produced

Other options include:

-   -   The user prints with a border at step 797 by selecting print        options from the menu before printing and selects to print with        a border, and the user can then select print from the        applications menu.    -   The user requests the date is printed with the photo at step 798        by selecting print options from the menu and selecting the print        date and time option, and the user can then select print from        the applications menu.    -   The user adds a caption at step 799 where the user selects photo        options from the menu and selects add caption,    -   wherein the user is prompted to enter a caption and the user        selects print from the applications menu.        7.1.2 Printing a Photo Archives the Photo

Referring to FIG. 70, printing a photo generally implies that the photohas more worth to the user than the photos the user has taken and notprinted, it is more likely to be shared and hence more likely to bereferred to in an archive. This scenario describes pushing a photo thatis printed to the photo archive on the mobile device 100 to give itpriority in being transferred off the device 100 and into the archive800.

When the photo is pushed to the archive the user is given theopportunity to specify what access permission's should be applied to thephoto at step 801 in the archive 800. In this scenario it is kept topublic or private 802 for simplicity, but it could easily be a morecomplex selection from various ACLs (Access Control Lists) maintained bythe user.

-   -   1. The user uses the default photo application 796 on their        device 100 and takes a photo.    -   2. The user may set some print options 801    -   3. The user selects print from the applications menu 803    -   4. The user is prompted whether the photo should be public or        private in the archive 802    -   5. The user is prompted to load a blank    -   6. The print is produced.

Optionally, default settings may be applied for archiving the imageswhere the user may not be prompted if the default settings indicatewhether all photos should be public or private.

This sequence diagram shows the most likely case, where uploading thephoto to the archive occurs after printing has finished. This is themost likely case, since power demands of printing require the networkconnectivity section of the phone to be powered down, or at leastavoided. Some devices may be able to support both, in which case theupload could occur during the print. The photo archive is accessiblefrom both a browser on a mobile device 100 or a desktop browser. Theuser is able to print a photo from the archive to either a desktopprinter or the printer in their mobile device.

7.1.3 Printing Archives a Netpage Enabled Photo

In this use case the user interactions are the same as for “Printing aPhoto Archives the Photo”, but in the background, the document, theprintout, and the impression ID are registered with the Netpageinfrastructure. The printout can be interacted with via a Netpagepointer 533 immediately on the device it was printed on, but it is notbe active for other pointers or pens until it has been successfullyuploaded to the network based Netpage services.

Referring to FIG. 71, for Netpage enabled photos that already are beingarchived in a general purpose photo archive it is not desirable for theNetpage server 529 to also store the photo, so the Netpage server 529stores a reference to the photo in the archive 800, allowing it to beretrieved when necessary. The Netpage server 529 still tracks userinteractions with the photo: reprints, digital ink, etc but it generallydoes not store the actual image itself. When the photo is pushed to thearchive the pusher receives a reference to the photo that can be used toretrieve the photo when it is required.

The need to move the photo and the Netpage document associations outinto the network to enable general Netpage interactive on the printedphoto, gives the pushing of the photo to the photo archive moreimportance. In this case the push to the archive includes a flagindicating the photo should be moved to the off device archive as soonas possible. The registering of a printout with the Netpage services cannot complete until the blank has been validated since the ID is used asthe Netpage impression ID.

In this case the photo reference rather than the photo is lodged withthe Netpage microserver. The printout can be interacted with via aNetpage pointer on the same device after the ID and signature have beenvalidated, but other pointers and pens can only interact with thedocument after it has been successfully uploaded to the Netpage networkserver.

8. Player Use Cases

8.1 Business Card Application

This section covers the Netpage Player PlayRequests 521 used in theinteractive Netpage Business Card and describes the behaviour of thePlayer Agents 537 handling such objects and commands. FIG. 72 shows thesample business card 810. The fields (text 811 and images 812) aregenerally associated with PlayRequests 521 that (partially) specify whataction is required when a user clicks on one of the fields with aNetpage pointer 533.

8.1.1 Phone Number

Referring to FIG. 73, the phone number fields 817, 818 on the businesscard 810 are associated with PlayRequests 816, 819 that simply specify aphone number, without specifying the operation to be performed or thetarget of the operation. The author of the business card 810 is thusproviding maximum freedom to the receiver of the business card 810 tomake use of the phone number fields as they see fit. For example, oneuser may have their system configured to react to such PlayRequests 816,819 by having their mobile phone dial the specified number, whileanother user may prefer to have such PlayRequests 816, 819 simply pushedto the Netpage Clipboard 615 for later use.

The mobile phone icon 813 is configured with a more fully specifiedPlayRequest 809 than the phone number fields 817, 818. The PlayRequest809 specifies an operation (“dial”) that should be performed when thatfield is selected. The operation overrides the default handling ofphone-number values that might otherwise be performed in the absence ofan explicit operation. Note that the target of the request is still leftunspecified. This gives the routing system the freedom to determine themost appropriate device with which to make the call. This is especiallyappropriate for a business card 810 which might be handed out tohundreds of users who each typically have their own phone. Placing aspecific target in the PlayRequest would have been possible, althoughinappropriate in this case. The landline telephone icon 813 isconfigured similarly to the mobile phone icon, but with a differentvalue for the phone number.

8.1.2 Fax Number

Referring to FIG. 74, the Fax phone number field 820 is configuredexactly as for the phone number fields except that the phone numbervalue is now also marked as belonging to the “fax” category 823. Thisadditional information can potentially be used during request routing inorder to select targets which specifically cater for “fax” phone numbersrather than targets which simply specify that they cater for phonenumbers in general. The “fax” button field 821 specifies an operation(“fax”) in the PlayRequest 822. The exact semantics of that operationare target dependant. For example, on a mobile phone, the Fax Agentmight launch the Fax editor with the destination fax numberpre-configured.

8.1.3 Web Address

Referring to FIG. 75, the web URL field 824 and “WWW” icon 825 areconfigured similarly to the phone number field and phone icons. The icon825 is specifically configured to cause a web browser to be invoked 826on the specified URL, whereas the PlayRequest 827 for the URL field isless fully specified and is therefore more flexible in terms of itspossible interpretations.

8.1.4 SMS and MMS Fields

Referring to FIG. 76, the “SMS” field 829 is configured to invoke an“SMS” operation 831. This request could be routed to an SMS Agentrunning on the user's mobile phone which would launch the SMS editorwith the destination phone number filled in. The “MMS” field 828 isconfigured similarly to the “SMS” field.

8.1.5 Email Address

Referring to FIG. 77, the web email address field 833 and “Email” icon832 are configured similarly to the phone number field and phone icons.The icon 832 is specifically configured to perform a “create-email”operation 834, which can typically be handled routing the request to anagent which is capable of launching email tool with the destinationemail address pre-configured. The email address field 833 is less fullyspecified and is therefore more flexible in terms of its possibleinterpretations.

8.1.6 Street Address

Referring to FIG. 78, the Street Address field 836 is configured to mapto a PlayRequest 837 that contains a location value 841 specified usinga type of location. The specific details of this type are not specifiedin this document. The important thing to note is that it storesinformation that specifies the location in various ways. As such, thevalue can be handled differently by a large number of agents. Examplesmight be:

-   -   A Web MAP Search Agent which presents the location in a web        browser by accessing a web-based map search facility    -   A Print Agent on an m-print phone which prints the location        details (and possibly directions) on an m-print card.    -   A GPS Navigator Agent which displays the location in a handheld        GPS device.        8.1.7 Photo

Referring to FIG. 79, the Name field 838 and “Photo” icon 839 areconfigured to map to a PlayRequest 840 that specifies a jpeg photograph.Typically, this request can be routed to an agent capable of displayingthe image.

8.2 Scanning Support

Referring to FIG. 80, interactive Netpage documents can be authored tospecify an action upon reception of a scan event. A scan event simplycontains the ID of the printed document without any {x,y} coordinateinformation. In the m-print context, a scan is achieved by feeding aprinted card back through the paper feed mechanism. This enables them-print device to determine the ID of the card and to transmit ascan-hit to the Netpage Server 529. Upon reception of a scan-hit, theserver 529 invokes any scan action that has been registered for theidentified printout.

The printed business card 820 can be authored to invoke a PlayRequest851 in response to a scan event. Typically, such a PlayRequest 851provides information pertaining to the entire content of the printout.In the case of the business card, the document can be authored to invokea PlayRequest 851 that contains a text/directory object 852 thatspecifies most of the details from the card. Such a PlayRequest 851would typically be routed to a Contacts Application (such as vCardAgent) which would then modify its database to include the details fromthe business card.

8.2.1 M-Print Photo Cards including Scan Support

FIG. 81 shows a printed photo card 855. The card 855 can be made to beinteractive as shown in FIG. 82. The card 855 contains two fields. Thefirst field is a standard Netpage field 856, while the second is thescan event field 857. Both fields 856, 857 are configured to map to aPlayRequest 858 that contains the contents of the photo in electronicform (in this case as a jpeg image). Typically, this request 858 isrouted to an agent capable of displaying the image, although the factthat the request is only partially specified (the target and operationfield are empty) gives the request router (and therefore the user) morefreedom to interpret the request as appropriate. For example, previousactions by the user may mean that clicking on the photo is interpretedas a request to associate the photo with a particular location onprinted Netpage document.

8.3 Printout Interactivity on the Mobile Device GUI

When a scan of an M-Print printout is performed on a mobile device 100the action taken can vary. One possible action is to display theprintout on the mobile device's GUI with the hyperlinks and form fieldsactive so that a user can navigate between them and fill them in oractivate them, in the same way a Netpage user can on a printout using aNetpage pen or pointer.

If the mobile device has a touch screen and a stylus then it is possibleto support all the interactions that a user with a Netpage pen couldhave with the printout. If the mobile device doesn't support a touchscreen and stylus then it is possible to tab through the hyperlinks andsubmit fields on the form and activate them in the same way that a userwith Netpage pointer and a printout could.

FIG. 83 shows a mobile phone 100 where an M-Print printout of a businesscard 820 has been scanned and the printout is now displayed on thephone's screen 860 with the first hyperlink 861 highlighted ready to beselected. The user can move through the active areas using standardnavigation keys 862 on the mobile device 100, in the same way they cannavigate the hyperlinks on a web page. Selecting a hyperlink in this wayvia the GUI is the same as clicking a Netpage Pointer 533 on theprintout of the business card 820.

Below is a use case illustrating the sequence of events for a user toactivate a hyperlink by scanning the printout.

8.3.1 Scan a Business Card and Send an MMS to the Person

-   -   1. The user inserts a printout into the paper feed slot on the        mobile device (870)    -   2. The printout is drawn through the mobile device    -   3. The image on the printout is displayed on the mobile device        (880), the first hyperlink has a focus region drawn around it    -   4. The user moves focus to the MMS hyperlink and selects it    -   5. The MMS editor on the device is displayed with the address        filled out from the details on the business card (890).

The upper portion of the sequence diagram in FIG. 84 is a repeat of thesequence for the case where a scan is occurring. When the NetpageMicroserver receives the “scan click” it retrieves the document anddisplays it on the screen using the Document Displayer. The DocumentDisplayer allows the user to step through the hyperlinks and fields andselect them. When one is selected the Netpage Microserver is sent aclick event, just as if it had come from a Netpage pointer. The Netpageprocessing results in a play request being sent to the Netpage Player onthe device, which responds by opening up the MMS editor with the messagealready addressed to the recipient identified by the business card thatwas scanned in.

9.0 Mobile Telecommunications Device Overview

Whilst the main embodiment includes both Netpage and printingfunctionality, only one or the other of these features is provided inother embodiments.

One such embodiment is shown in FIG. 85, in which a mobiletelecommunications device in the form of a mobile phone 1 (also known asa “cellphone”) includes a mobile phone module 2 and a printer module 4.The mobile 30 phone module is configured to send and receive voice anddata via a telecommunications network (not shown) in a conventionalmanner known to those skilled in the art. The printer module 4 isconfigured to print a page 6. Depending upon the particularimplementation, the printer module 4 can be configured to print the page6 in color or monochrome.

The mobile telecommunications device can use any of a variety of knownoperating systems, such as Symbian (with UIQ and Series 60 GUIs),Windows Mobile, PalmOS, and Linux.

In the preferred embodiment (described in more detail below), the printmedia is pre-printed with tags, and the printer module 4 prints visibleinformation onto the page 6 in registration with the tags. In otherembodiments, Netpage tags are printed by the printer module onto thepage 6 along with the other information. The tags can be printed usingeither the same visible ink as used to print visible information, orusing an infrared or other substantially invisible ink.

The information printed by the printer module 4 can include user datastored in the mobile phone 1 (including 10 phonebook and appointmentdata) or text and images received via the telecommunications network orfrom another device via a communication mechanism such as Bluetooth™ orinfrared transmission. If the mobile phone 1 includes a camera, theprinter module 4 can be configured to print the captured images. In thepreferred form, the mobile phone module 2 provides at least basicediting capabilities to enable cropping, filtering or addition of textor other image data to the captured image before printing. Theconfiguration and operation of the printer module 4 is described in moredetail below in the context of various types of mobile telecommunicationdevice that incorporate a printhead.

FIG. 86 shows another embodiment of a mobile telecommunications device,in which the printer module 4 is omitted, and a Netpage tag sensormodule 8 is included. The Netpage module 8 enables interaction betweenthe mobile phone 1 and a page 10 including Netpage tags. Theconfiguration and operation of the Netpage pointer in a mobile phone 1is described in more detail below. Although not shown, the mobile phone1 with Netpage module 8 can include a camera.

FIG. 87 shows a mobile phone 1 that includes both a printer module 4,and a Netpage tag sensor module 8. As with the embodiment of FIG. 86,the printer module 4 can be configured to print tagged or untaggedpages. As shown in FIG. 87, where tagged pages 10 are produced (andirrespective of whether the tags were pre-printed or printed by theprinter module 4), the Netpage tag sensor module 8 can be used tointeract with the resultant printed media.

A more detailed architectural view of the mobile phone 1 of FIG. 87 isshown in FIG. 88, in which features corresponding to those shown in FIG.87 are indicated with the same reference numerals. It will beappreciated that FIG. 88 deals only with communication between variouselectronic components in the mobile telecommunications device and omitsmechanical features. These are described in more detail below.

The Netpage tag sensor module 8 includes a monolithically integratedNetpage image sensor and processor 12 that captures image data andreceives a signal from a contact switch 14. The contact switch 14 isconnected to a nib (not shown) to determine when the nib is pressed intocontact with a surface. The sensor and processor 12 also outputs asignal to control illumination of an infrared LED 16 in response to thestylus being pressed against the surface. The image sensor and processor12 outputs processed tag information to a Netpage pointer driver 18 thatinterfaces with the phone operating system 20 running on the mobiletelecommunications device's processor (not shown).

Output to be printed is sent by the phone operating system 20 to aprinter driver 22, which passes it on to a MoPEC chip 24. The MoPEC chipprocesses the output to generate dot data for supply to the printhead26, as described in more detail below. The MoPEC chip 24 also receives asignal from a media sensor 28 indicating when the media is in positionto be printed, and outputs a control signal to a media transport 30.

The printhead 26 is disposed within a replaceable cartridge 32, whichalso includes ink 34 for supply to the printhead.

9.1 Mobile Telecommunications Device Module

FIG. 89 shows the mobile phone module 2 in more detail. The majority ofthe components other than those directly related to printing and Netpagetag sensing are standard and well known to those in the art. Dependingupon the specific implementation of the mobile phone 1, any number ofthe illustrated components can be included as part of one or moreintegrated circuits.

Operation of, and communication between, the mobile phone module 2components is controlled by a mobile phone controller 36. The componentsinclude:

-   -   'mobile radio transceiver 38 for wireless communication with a        mobile telecommunications network;    -   program memory 40 for storing program code for execution on the        mobile phone controller 36;    -   working memory 42 for storing data used and generated by the        program code during execution. Although shown as separate from        the mobile phone controller 36, either or both memories 40 and        42 may be incorporated in the package or silicon of the        controller;    -   keypad 44 and buttons 46 for accepting numerical and other user        input;    -   touch sensor 48 which overlays display 50 for accepting user        input via a stylus or fingertip pressure;    -   removable memory card 52 containing non-volatile memory 54 for        storing arbitrary user data, such as digital photographs or        files;    -   local area radio transceiver 56, such as a Bluetoothrm        transceiver;    -   GPS receiver 58 for enabling determination of the location of        the mobile telecommunications device (alternatively the phone        may rely on mobile network mechanisms for determining its        location);    -   microphone 60 for capturing a user's speech;    -   speaker 62 for outputting sounds, including voice during a phone        call;    -   camera image sensor 64 including a CCD for capturing images;    -   camera flash 66;    -   power manager 68 for monitoring and controlling power        consumption of the mobile telecommunications device and its        components; and    -   SIM (subscriber Identity Module) card 70 including SIM 72 for        identifying the subscriber to mobile networks.

The mobile phone controller 36 implements the baseband functions ofmobile voice and data communications protocols such as GSM, GSM modemfor data, GPRS and CDMA, as well as higher-level messaging protocolssuch as SMS and MMS. The one or more local-area radio transceivers 56enable wireless communication with peripherals such as headsets andNetpage pens, and hosts such as personal computers. The mobile phonecontroller 36 also implements the baseband functions of local-area voiceand data communications protocols such as IEEE 802.11, IEEE 802.15, andBluetooth™.

The mobile phone module 2 may also include sensors and/or motors (notshown) for electronically adjusting zoom, focus, aperture and exposurein relation to the digital camera.

Similarly, as shown in FIG. 90, components of the printer module 4include:

-   -   print engine controller (PEC) 74 in the form of a MoPEC device;    -   program memory 76 for storing program code for execution by the        print engine controller 74;    -   working memory 78 for storing data used and generated by the        program code during execution by the print engine controller 74;        and    -   a master QA chip 80 for authenticating printhead cartridge 32        via its QA chip 82.

Whilst the printhead cartridge in the preferred form includes the inksupply 34, the ink reservoirs can be housed in a separate cartridge inalternative embodiments.

FIG. 91 shows the components of the tag sensor module 8, which includesa CMOS tag image processor 74 that communicates with image memory 76. ACMOS tag image sensor 78 sends captured image data to the processor 74for processing. The contact sensor 14 indicates when a nib (not shown)is brought into contact with a surface with sufficient force to close aswitch within the contact sensor 14. Once the switch is closed, theinfrared LED 16 illuminates the surface, and the image sensor 78captures at least one image and sends it to the image processor 74 forprocessing. Once processed (as described below in more detail), imagedata is sent to the mobile phone controller 36 for decoding.

In an alternative embodiment, shown in FIG. 92, the tag sensor module 8is replaced by a tag decoder module 81. The tag decoder module 81includes all the elements of the tag sensor module 8, but adds ahardware-based tag decoder 82, as well as program memory 84 and workingmemory 86 for the tag decoder. This arrangement reduces thecomputational load placed on the mobile phone controller, with acorresponding increase in chip area compared to using the tag sensormodule 8.

The Netpage sensor module can be incorporated in the form of a Netpagepointer, which is a simplified Netpage pen suitable mostly foractivating hyperlinks. It preferably incorporates a non-marking stylusin place of the pen's marking nib (described in detail later in thespecification); it uses a surface contact sensor in place of the pen'scontinuous force sensor; and it preferably operates at a lower positionsampling rate, making it unsuitable for capturing drawings andhand-writing. A Netpage pointer is less expensive to implement than aNetpage pen, and tag image processing and tag decoding can potentiallybe performed by software without hardware support, depending on samplingrate.

The various aspects of the invention can be embodied in any of a numberof mobile telecommunications device types. Several different devices aredescribed here, but in the interests of brevity, the detaileddescription will concentrate on the mobile telecommunications deviceembodiment.

9.2 Mobile Device

One preferred embodiment is the non-Netpage enabled ‘candy bar’ mobiletelecommunications device in the form of a mobile phone shown in FIGS.92 to 98. While a candy bar style phone is described here, it couldequally take the form of a “flip” style phone, which includes a pair ofbody sections that are hinged to each other. Typically, the display isdisposed on one of the body sections, and the keypad is disposed on theother, such that the display and keypad are positioned adjacent to eachother when the device is in the closed position.

In further embodiments, the device can have two body sections thatrotate or slide relative to each other. Typically, the aim of thesemechanical relationships between first and second body sections is toprotect the display from scratches and/or the keypad from accidentalactivation. Photo printing is considered one of the most compelling usesof the mobile Memjet printer. A preferred embodiment of the inventiontherefore includes a camera, with its attendant processing power andmemory capacity.

The elements of the mobile telecommunications device are best shown inFIG. 93, which (for clarity) omits minor details such as wires andhardware that operatively connect the various elements of the mobiletelecommunications device together. The wires and other hardware will bewell known to those skilled in the art. The mobile phone 100 comprises achassis moulding 102, a front moulding 104 and a rear cover moulding106. A rechargeable battery 108, such as a lithium ion or nickel metalhydride battery, is mounted to the chassis moulding 102 and covered bythe rear cover moulding 106. The battery 108 powers the variouscomponents of the mobile phone 100 via battery connector 276 and thecamera and speaker connector 278.

The front moulding 104 mounts to the emphasis to enclose the variouscomponents, and includes numerical interface buttons 136 positioned invertical rows on each side of the display 138. A multi-directionalcontrol pad 142 and other control buttons 284 enable menu navigation andother control inputs. A daughterboard 280 is mounted to the chassismoulding 102 and includes a directional switch 286 for the multidirectional control pad 142. The mobile telecommunications deviceincludes a cartridge access cover 132 that protects the interior of themobile telecommunications device from dust and other foreign objectswhen a print cartridge 148 is not inserted in the cradle 124.

An optional camera module 110 is also mounted to the chassis moulding102, to enable image capture through a hole 112 in the rear covermoulding 106. The camera module 110 includes a lens assembly and a CCDimage sensor for capturing images. A lens cover 268 in the hole 112protects the lens of the camera module 110. The rear cover moulding 106also includes an inlet slot 228 and an outlet slot 150 through whichprint media passes.

The chassis moulding 102 supports a data/recharge connector 114, whichenables a proprietary data cable to be plugged into the mobiletelecommunications device for uploading and downloading data such asaddress book information, photographs, messages, and any type ofinformation that might be sent or received by the mobiletelecommunications device. The data/recharge connector 114 is configuredto engage a corresponding interface in a desktop stand (not shown),which holds the mobile telecommunications device in a generally uprightposition whilst data is being sent or received by the mobiletelecommunications device. The data/recharge connector also includescontacts that enable recharging of the battery 108 via the desktopstand. A separate recharge socket 116 in the data/recharge connector 114is configured to receive a complimentary recharge plug for enablingrecharging of the battery when the desktop stand is not in use.

A microphone 170 is mounted to the chassis moulding 102 for convertingsound, such as a user's voice, into an electronic signal to be sampledby the mobile telecommunications device's analog to digital conversioncircuitry. This conversion is well known to those skilled in the art andso is not described in more detail here. A SIM (Subscriber IdentityModule) holder 118 is formed in the chassis moulding 102, to receive aSIM card 120. The chassis moulding is also configured to support a printcartridge cradle 124 and a drive mechanism 126, which receive areplaceable print cartridge 148. These features are described in moredetail below. Another moulding in the chassis moulding 102 supports anaerial (not shown) for sending and receiving RF signals to and from amobile telecommunications network.

A main printed circuit board (PCB) 130 is supported by the chassismoulding 102, and includes a number of momentary pushbuttons 132. Thevarious integrated and discrete components that support thecommunications and processing (including printing processing) functionsare mounted to the main PCB, but for clarity are not shown in thediagram.

A conductive elastomeric overlay 134 is positioned on the main PCB 130beneath the keys 136 on the front 40 moulding 104. The elastomerincorporates a carbon impregnated pill on a flexible profile. When oneof the keys 136 is pressed, it pushes the carbon pill to a 2-wire opencircuit pattern 132 on the PCB surface. This provides a low impedanceclosed circuit. Alternatively, a small dome is formed on the overlaycorresponding to each key 132.

Polyester film is screen printed with carbon paint and used in a similarmanner to the carbon pills. Thin adhesive film with berrylium copperdomes can also be used. A loudspeaker 144 is installed adjacentapertures 272 in the front moulding 104 to enable a user to hear soundsuch as voice communication and other audible signals.

A color display 138 is also mounted to the main PCB 130, to enablevisual feedback to a user of the mobile telecommunications device. Atransparent lens moulding 146 protects the display 138. In one form, thetransparent lens is touch-sensitive (or is omitted and the display 138is touch sensitive), enabling a user to interact with icons and inputtext displayed on the display 138, with a finger or stylus.

A vibration assembly 274 is also mounted to the chassis moulding 102,and includes a motor that drives an eccentrically mounted weight tocause vibration. The vibration is transmitted to the chassis 102 andprovides tactile feedback to a user, which is useful in noisyenvironments where ringtones are not audible.

10. Print Media Printing

A Netpage printer normally prints the tags which make up the surfacecoding on demand, i.e. at the same time as it prints graphic pagecontent. As an alternative, in a Netpage printer not capable of printingtags such as the preferred embodiment, pre-tagged but otherwise blankNetpages can be used. The printer, instead of being capable of tagprinting, typically incorporates a Netpage tag sensor. The printersenses the tags and hence the region ID of a blank either prior to,during, or after the printing of the graphic page content onto theblank. It communicates the region ID to the Netpage server, and theserver associates the page content and the region ID in the usual way.

A particular Netpage surface coding scheme allocates a minimum number ofbits to the representation of spatial coordinates within a surfaceregion. If a particular media size is significantly smaller than themaximum size representable in the minimum number of bits, then theNetpage code space may be inefficiently utilised. It can therefore be ofinterest to allocate different sub-areas of a region to a collection ofblanks. Although this makes the associations maintained by the Netpageserver more complex, and makes subsequent routing of interactions morecomplex, it leads to more efficient code space utilisation. In the limitcase the surface coding may utilise a single region with a singlecoordinate space, i.e. without explicit region IDs.

If regions are sub-divided in this way, then the Netpage printer usesthe tag sensor to determine not only the region ID but also the surfacecoding location of a known physical position on the print medium, i.e.relative to two edges of the medium. From the surface coding locationand its corresponding physical position on the medium, and the known (ordetermined) size of the medium, it then determines the spatial extent ofthe medium in the region's coordinate space, and communicates both theregion ID and the spatial extent to the server. The server associatesthe page content with the specified sub-area of the region.

A number of mechanisms can be used to read tag data from a blank. Aconventional Netpage tag sensor incorporating a two-dimensional imagesensor can be used to capture an image of the tagged surface of theblank at any convenient point in the printer's paper path. As analternative, a linear image sensor can be used to capture successiveline images of the tagged surface of the blank during transport. Theline images can be used to create a two-dimensional image which isprocessed in the usual way. As a further alternative, region ID data andother salient data can be encoded linearly on the blank, and a simplephotodetector and ADC can be used to acquire samples of the linearencoding during transport.

One important advantage of using a two-dimensional image sensor is thattag sensing can occur before motorised transport of the print mediumcommences. For example, if the print medium is manually inserted by theuser, then tag sensing can occur during insertion. This has the furtheradvantage that if the tag data is validated by the device, then theprint medium can be rejected and possibly ejected before printingcommences. For example, the print medium may have been pre-printed withadvertising or other graphic content on the reverse side from theintended printing side. The device can use the tag data to detectincorrect media insertion, i.e. upside-down or back-to-front. The devicecan also prevent accidental overprinting of an already-printed medium.And it can detect the attempted use of an invalid print medium andrefuse printing, eg. to protect print quality. The device can alsoderive print medium characteristics from the tag data, to allow it toperform optimal print preparation.

If a linear image sensor is used, or if a photodetector is used, thenimage sensing must occur during motorised transport of the print mediumto ensure accurate imaging. Unless there are at least two points ofcontact between the transport mechanism and the print medium in theprinting path, separated by a minimum distance equal to the tag dataacquisition distance, tag data cannot be extracted before printingcommences, and the validation advantages discussed above do not obtain.In the case of a linear image sensor, the tag data acquisition distanceequals the diameter of the normal tag imaging field of view. In the caseof a photodetector, the tag data acquisition distance is as long as therequired linear encoding.

If the tag sensor is operable during the entire printing phase at asufficiently high sampling rate, then it can also be used to performaccurate motion sensing, with the motion data being used to provide aline synchronisation signal to the print engine. This can be used toeliminate the effects of jitter in the transport mechanism.

FIGS. 100 to 106 show one embodiment of the encoded medium and the mediasensing and printing system within the mobile telecommunications device.While the encoding of the cards is briefly discussed here, it isdescribed in detail in the Coded Media sub-section of thisspecification.

Referring to FIG. 100, the ‘back-side’ of one of the cards 226 is shown.The back-side of the card has two coded data tracks: a ‘clock track’ 434and a ‘data track’ 436 running along the longitudinal sides of thecards. The coded data may be in the form of a two-dimensional grid orpattern. The cards are encoded with data indicating, inter alia:

-   -   the orientation of the card;    -   the media type and authenticity;    -   the longitudinal size;    -   the pre-printed side;    -   detection of prior printing on the card; and,    -   the position of the card relative to the printhead IC.

In one form, the encoded data is printed in IR ink so that it isinvisible and does not encroach on the space available for printingvisible images.

In a basic form, the M-Print cards 226 are only encoded with a datatrack and clocking (as a separate clock track or a self-clocking datatrack). However, in the more sophisticated embodiment shown in thefigures, the cards 226 have a pre-printed Netpage tag pattern 438covering the majority of the back-side. The front side may also have apre-printed tag pattern. It is preferred in these embodiments that thedata track encodes first information that is at least indicative ofsecond information encoded in the tags. Most preferably, the firstinformation is simply the document identity that is encoded in each ofthe tags.

The clock track 434 allows the MoPEC 326 (see FIG. 101) to determine, byits presence, that the front of the card 226 is facing the printhead202, and allows the printer to sense the motion of the card 226 duringprinting. The clock track 434 also provides a clock for the denselycoded data track 436.

The data track 436 provides the Netpage identifier and optionallyassociated digital signatures which allows MoPEC 326 to rejectfraudulent or un-authorised media 226, and to report the Netpageidentifier of the front-side Netpage tag pattern to a Netpage server. Itshould be noted that a fragment of a digital signature can also beconsidered a digital signature in its own right.

FIG. 101 shows a block diagram of an M-Print system that uses mediaencoded with separate clock and data tracks. The clock and data tracksare read by separate optical encoders. The system may optionally have anexplicit edge detector 474 which is discussed in more detail below inrelation to FIG. 104.

FIG. 102 shows a simplified circuit for an optical encoder which may beused as the clock track or data track optical encoder. It incorporates aSchmitt trigger 466 to provide the MoPEC 326 with an essentially binarysignal representative of the marks and spaces encountered by the encoderin the clock or data track. An IR LED 472 is configured to illuminate amark-sized area of the card 226 and a phototransistor 468 is configuredto capture the light 470 reflected by the card. The LED 472 has a peakwavelength matched to the peak absorption wavelength of the infrared inkused to print the media coding.

As an alternative, the optical encoders can sense the direction of mediamovement by configuring them to be ‘quadrature encoders’. A quadratureencoder contains a pair of optical encoders spatially positioned to readthe clock track 90 degrees out of phase. Its in-phase and quadratureoutputs allow the MoPEC 326 to identify not just the motion of the clocktrack 434 but also the direction of the motion. A quadrature encoder isgenerally not required, since the media transport direction is known apriori because the printer controller also controls the transport motor.However, the use of a quadrature encoder can help decouple abi-directional motion sensing mechanism from the motion controlmechanism.

FIG. 103 shows a block diagram of the MoPEC 326. It incorporates adigital phase lock loop (DPLL) 444 to track the clock inherent in theclock track 434 (see FIG. 100), a line sync generator 448 to generatethe line sync signal 476 from the clock 446, and a data decoder 450 todecode the data in the data track 436. De-framing, error detection anderror correction may be performed by software running on MoPEC'sgeneral-purpose processor 452, or it may be performed by dedicatedhardware in MoPEC.

The data decoder 450 uses the clock 446 recovered by the DPLL 444 tosample the signal from the data track optical encoder 442. It may eithersample the continuous signal from the data track optical encoder 442, orit may actually 5 trigger the LED of the data track optical encoder 442for the duration of the sample period, thereby reducing the total powerconsumption of the LED. The DPLL 444 may be a PLL, or it may simplymeasure and filter the period between successive clock pulses.

The line sync generator 456 consists of a numerically-controlledoscillator which generates line sync pulses 476 at a rate which is amultiple of the rate of the clock 446 recovered from the clock track434.

As shown in FIG. 101, the print engine may optionally incorporate anexplicit edge detector 474 to provide longitudinal registration of thecard 226 with the operation of the printhead 202. In this case, as shownin FIG. 104, it generates a page sync signal 478 to signal the start ofprinting after counting a fixed number of line syncs 476 after edgedetection. Longitudinal registration may also be achieved by othercard-in detection mechanisms ranging from opto-sensors, de-cappingmechanical switches, drive shaft/tension spring contact switch and motorload detection.

Optionally, the printer can rely on the media coding itself to obtainlongitudinal registration. For example, it may rely on acquisition of apilot sequence on the data track 436 to obtain registration. In thiscase, as shown in FIG. 105, it generates a page sync signal 478 tosignal the start of printing after counting a fixed number of line syncs476 after pilot detection. The pilot detector 460 consists of a shiftregister and combinatorial logic to recognise the pilot sequence 480provided by the data decoder 450, and generate the pilot sync signal482. Relying on the media coding itself can provide superior informationfor registering printed content with the Netpage tag pattern 438 (FIG.100).

As shown in FIG. 106, the data track optical encoder 442 is positionedadjacent to the first clock data encoder 440, so that the data track 436(see FIG. 100) can be decoded as early as possible and using therecovered clock signal 446. The clock must be acquired before printingcan commence, so a first optical encoder 440 is positioned before theprinthead 202 in the media feed path. However, as the clock needs to betracked throughout the print, a second clock optical encoder 464 ispositioned coincident with or downstream of the printhead 202. This isdescribed in more detail below.

FIG. 99 shows the printed card 226 being withdrawn from the printcartridge, 148. It will be appreciated that the printed card 226 needsto be manually withdrawn by the user. Once the trailing edge of the card226 has passed between the drive shaft 178 and the spring fingers 238,it is no longer driven along the media feed path. However, as theprinthead 202 is less than 2 mm from the drive shaft 178, the momentumof the card 226 projects the trailing edge of past the printhead 202.

While the momentum of the card is sufficient to carry the trailing edgepast the printhead, it is not enough to fling it out of the exit slot150 (FIG. 98). Instead, the card 226 is lightly gripped by the opposedlock actuator arms 232 as it protrudes from the exit slot 150 in theside of the mobile phone 100. This retains the card 226 so it does notsimply fall from exit slot 150, but rather allows users to manuallyremove the printed card 226 from the mobile phone 100 at theirconvenience. This is important to the practicality of the mobiletelecommunications device because the card 226 is fed into one side ofthe mobile telecommunications device and retrieved from the other, sousers will typically want to swap the hand that holds the mobiletelecommunications device when collecting the printed card. By lightlyretaining the printed card, users do not need to swap hands and be readyto collect the card before completion of the print job (approximately1-2 secs). Alternatively, the velocity of the card as it leaves theroller can be made high enough that the card exits the outlet slot 123under its own inertia.

10.1 M-Print Flip Printing

One can allow a previously printed m-print Netpage card to bere-inserted into the printing mechanism “flipped-over”, so that the sidenot previously printed on (i.e. the back of the card) is now facingtowards the print head. The printer would detect such an insertion andwould automatically print additional information on to the back of thecard. The additional information would typically, but not necessarily,be application and context specific. That is:

-   -   the application which created the original printout would        determine what is printed onto the back side of the card, and    -   would be able to take into account context specific information        such as the impression ID of the card.

This allows applications to print information onto the back side of thecard which is specific to the original printout on the front side of thecard. There are many potential uses for such a mechanism. For the sakeof discussion, one such case is described below.

10.1.1 Camera Use-Case

-   -   1. User takes a photo using an m-print enabled camera phone    -   2. User prints photo onto Netpage tagged card    -   3. User feeds card back through printer flipped-over    -   4. Various details about the photo would then be printed onto        the back of the card. Examples of details might be:        -   The date and time the photo was taken;        -   Location where photo was taken, either automatically            determined by a geographical positioning system within the            phone (eg. GPS or cell-based location detection), or            manually entered after the fact by the user; and/or        -   Arbitrary textual information entered by the user (perhaps            entered on the phone itself or via a web-based photo            archiving application sometime after the photo was taken).

The time duration between the user taking the original photo andinserting the flipped-over card could be arbitrarily long. As such, themechanism can act as a “what is this photo that I just found?” facility.Another advantage is that it removes the need to have text obscuringparts of the photo in order to provide date/time information.

11. General Netpage Overview

Netpage interactivity can be used to provide printed user interfaces tovarious phone functions and applications, such as enabling particularoperational modes of the mobile telecommunications device or interactingwith a calculator application, as well as providing general “keypad”,“keyboard” and “tablet” input to the mobile telecommunications device.Such interfaces can be pre-printed and bundled with a phone, purchasedseparately (as a way of customizing phone operation, similar toringtones and themes) or printed on demand where the phone incorporatesa printer.

A printed Netpage business card provides a good example of how a varietyof functions can be usefully combined in a single interface, including:

-   -   loading contact details into an address book    -   displaying a Web page    -   displaying an image    -   dialing a contact number    -   bringing up an e-mail, SMS or MMS form    -   loading location info into a navigation system    -   activating a promotion or special offer

Any of these functions can be made single-use only. A business card maybe printed by the mobile telecommunications device user for presentationto someone else, or may be printed from a Web page relating to abusiness for the mobile telecommunications device user's own use. It mayalso be pre-printed.

As described below, the primary benefit of incorporating a Netpagepointer or pen in another device is synergy. A Netpage pointer or penincorporated in a mobile phone, smartphone or telecommunications-enabledPDA, for example, allows the device to act as both a Netpage pointer andas a relay between the pointer and the mobile phone network and hence aNetpage server. When the pointer is used to interact with a page, thetarget application of the interaction can display information on thephone display and initiate further interaction with the user via thephone touchscreen. The pointer is most usefully configured so that its“nib” is in a corner of the phone body, allowing the user to easilymanipulate the phone to designate a tagged surface. The phone canincorporate a marking nib and optionally a continuous force sensor toprovide full Netpage pen functionality.

An exemplary Netpage interaction will now be described to show how asensing device in the form of a Netpage enabled mobile device interactswith the coded data on a print medium in the form of a card. Whilst inthe preferred form the print medium is a card generated by the mobiledevice or another mobile device, it can also be a commerciallypre-printed card that is purchased or otherwise provided as part of acommercial transaction. The print medium can also be a page of a book,magazine, newspaper or brochure, for example. The print medium can beprovided with coded data in a variety of formats, the coded dataencoding a range of information, preferably, at least some of theinformation being indicative of the print media identifier. Theinformation can be indicative of a two-dimensional coordinate grid, andthe format can be a two-dimensional pattern.

For example, the print medium can be provided with first coded data in afirst format and second coded data in a second format, the first codeddata encoding first information and the second coded data encodingsecond information, with at least some of the first information beingindicative of the print media identifier, the first format being alinear pattern, and with at least some of the second information beingindicative of the print media identifier and of a two-dimensionalcoordinate grid, the second format being a two-dimensional pattern. In aparticular example form, the information is further indicative of atleast part of a digital signature associated with the print mediaidentifier, the sensor module determining, by reading at least some ofthe coded data, at least part of the digital signature, and the printermodule can then print, if the digital signature is authentic, content onthe print media.

The mobile device senses a tag using an area image sensor and detectstag data. The mobile device uses the sensed data tag to generateinteraction data, which is sent via a mobile telecommunications networkto a document server. The document server uses the ID to access thedocument description, and interpret the interaction. In appropriatecircumstances, the document server sends a corresponding message to anapplication server, which can then perform a corresponding action.

Typically Netpage pen and Netpage-enabled mobile device users registerwith a registration server, which associates the user with an identifierstored in the respective Netpage pen or Netpage enabled mobile device.By providing the sensing device identifier as part of the interactiondata, this allows users to be identified, allowing transactions or thelike to be performed. Netpage documents are generated by having an IDserver generate an ID which is transferred to the document server. Thedocument server determines a document description and then records anassociation between the document description and the ID, to allowsubsequent retrieval of the document description using the ID. The ID isthen used to generate the tag data, as will be described in more detailbelow, before the document is printed by a suitable printer, using thepage description and the tag map.

Each tag is represented by a pattern which contains two kinds ofelements. The first kind of element is a target. Targets allow a tag tobe located in an image of a coded surface, and allow the perspectivedistortion of the tag to be inferred. The second kind of element is amacrodot. Each macrodot encodes the value of a bit by its presence orabsence. The pattern is represented on the coded surface in such a wayas to allow it to be acquired by an optical imaging system, and inparticular by an optical system with a narrowband response in thenear-infrared. The pattern is typically printed onto the surface using anarrowband near-infrared ink.

In the preferred embodiment, the region typically corresponds to theentire surface of an M-Print card, and the region ID corresponds to theunique M-Print card ID. For clarity in the following discussion we referto items and IDs, with the understanding that the ID corresponds to theregion ID. The surface coding is designed so that an acquisition fieldof view large enough to guarantee acquisition of an entire tag is largeenough to guarantee acquisition of the ID of the region containing thetag. Acquisition of the tag itself guarantees acquisition of the tag'stwo-dimensional position within the region, as well as othertag-specific data. The surface coding therefore allows a sensing deviceto acquire a region ID and a tag position during a purely localinteraction with a coded surface, e.g. during a “click” or tap on acoded surface with a pen.

Optional embodiments of the present invention may also be said tobroadly consist in the parts, elements and features referred to orindicated herein, individually or collectively, in any or allcombinations of two or more of the parts, elements or features, andwherein specific integers are mentioned herein which have knownequivalents in the art to which the invention relates, such knownequivalents are deemed to be incorporated herein as if individually setforth.

Although a preferred embodiment has been described in detail, it shouldbe understood that various changes, substitutions, and alterations canbe made by one of ordinary skill in the art without departing from thescope of the present invention.

1. A method of printing an electronic document on a print medium using amobile telecommunications device, the method comprising the steps of:identifying the electronic document using the mobile telecommunicationsdevice; sensing coded data encoded on the print medium using a sensormodule of the mobile telecommunications device; determining from thecoded data a print media identifier which identifies the print medium;printing the electronic document on the print medium using a printermodule of the mobile telecommunications device, and forming anassociation between the print media identifier and the electronicdocument, such that the electronic document is retrievable using theprint media identifier.
 2. The method as claimed in claim 1, wherein theelectronic document is an image and is captured using a camera of themobile telecommunications device.
 3. The method as claimed in claim 1,wherein the print medium is provided with first coded data in a firstdata format, the first coded data encoding first information comprisingat least the print media identifier.
 4. The method as claimed in claim3, wherein the first data format is a linear pattern.
 5. The method asclaimed in claim 3, wherein the print medium is further provided withsecond coded data in a second data format, the second coded dataencoding second information comprising at least the print mediaidentifier and is indicative of a two-dimensional coordinate grid. 6.The method as claimed in claim 3, wherein the print medium is furtherprovided with second coded data in a second data format, the secondcoded data encoding second information comprising at least the printmedia identifier and is indicative of regions of a surface of the printmedium.
 7. The method as claimed in claim 5, wherein the second codeddata is provided on the print medium by the print module printing thesecond coded data onto the print medium.
 8. The method as claimed inclaim 6, wherein the electronic document is associated with a region ofthe print medium using the second coded data.
 9. The method as claimedin claim 5, wherein the electronic document is frirther associated witha particular location on the print medium.